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ABSTRACT 

This  paper  is  a  survey  of  existing  literature  describing 
software  quality  assurance  and  an  indepth  evaluation  of  both 
selected  industry  quality  a ssuranca  functions  and  the  Fleet 
Material  Support  Office  (F51S0)  Quality  Assurance  Division. 
Quality  control  at  FMSO  is  effected  by  the  orqanizational 
element  that  produces  the  product  aad  by  a  small,  central- 
iz3d  staff.  Improved  systems  development  and  a  higher  level 
of  quality  control  are  tha  goals  of  FMSO.  The  recommenda- 
tions and  conclusions  offered  are  based  on  an  axtensive 
literature  search  of  existing  material  on  software  quality 
assurance,  an  indepth  study  of  selected  industry  quality 
assurance  departments,  and  an  examination  of  the  current 
state  or  quality  control  procedures  at  FMSO,  These  recom- 
mendations, if  implemented,  should  serve  -c  improve  the 
quality  control  at  FMSO  and  assist  -he  organization  in 
achieving  their  goals. 
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I.   iNI^o^ncTio^ 

The  computer  industry  has  gradually  evol'/ed  ov^r  the 
years  from  the  massive  hardware  systems  that  were  very 
expensive  to  build  and  maintain  to  +-h3  present  state  of  the 
art  where  a  tiry  one  quarter  inch  square  chip  has  mors 
computing  power,  is  enormously  chaaper  to  buy,  and  can  be 
maintained   virtually    100      percent   of   the   time.  Along    with 

the  hardwar*=^  change  we  hav=  seen  an  iven  grea-cec  improvement 
in  "che  software  developmenc  from  sioiole  ma-hemarical  compu- 
tation to  the  launching  and  recovery  of  manned  space 
vehicles  with  greater  reliability  and  performance  than  at 
any    other   time    in  history. 

This  evolution  has  not  happened  by  acciden-^-.  The  devel- 
opment has  been  based  upon  making  mistakes,  documenting 
those  mistakes,  and  processing  on.  As  people  continued  the 
process  they  studied  past  documentation  and  continued  the 
dccumentaricn  process  until  it  has  become  an  accepted  part 
of   the  development    cycle.  Although    r.o    systematic   approach 

was  utilized,  there  has  been  an  avsr  increasing  tendency 
towards  the  development  of  a  set  of  standards  that  could 
provide  an  avenue  for  common  understanding  with  a  minimum  of 
confusion. 

The  growth  of  systems  software  which  occured  during  the 
pasx  two  decades  has  presented  each  organization  with 
continuing  challenges  in  maintaining  an  effective  organiza- 
tional structure  and  in  following  efficient  systems 
development  methods  and  strategies  which  can  be  transferred 
from  one  entity  to  another  with  a  maximum  degree  of  cohe- 
siveness  and  precision  a  nd  a  minimum  '"-f  ambiguity  and 
con  fusion. 


Acccrdir.g  zo  D-  Ross  of  SofTach  Inc.  [Rsf.  1],  tha 
Q'U2.!Li."iy  o£  sof'TiwsZ'S  z.s  r9l.2.Ti.vs  "to  ths  i-r/'i'^ndsd  sot^Ilc^t  ion 
and  can  only  be  achieved  by  a  disciplined  methodology  in 
which  quality  requirements  are  initially  applied  to  the 
original  requirements  definition  for  the  problem  and  are 
than  carefully  checked  and  confirmed  at  every  stage  in  the 
production  process.  In  order  to  have  any  sensible  treatment 
of  quality  every  aspect  of  the  system  life  cycle  must  be 
based  on  an  orderly,  controlled,  ind  disciplined  metho- 
dology. This  is  not  to  say  that  all  software  must  be 
produced  exactly  with  the  same  set  of  -ools  and  techniques, 
for  this  clearly  would  be  excessive.  Ther'^  m.av  be  a  broad 
spectrum  of  the  degree  to  which  the  ideal  system  technology 
is  approached,  but  even  the  simplified,  streamlined  metho- 
dologies must  be  complete  and  consistent.  Each  version  must 
be,  in  seme  reasonable  fashion,  a  proper  degeneration  of  the 
elaborate,  most  advanced  state  of  the  art,  merely  simplified 
to  suit  a  simpler  set  of  circumstanoes.  It  mast  be  incum- 
bent upon  all  persons  that  are  engaged  in  the  project  no 
combine  company  standards  and  common  sense  to  arrive  at  the 
required  level  t ha~  will  ensure  a  gaali-y  product. 

To  accomplish  this,  there  exists  within  each  organiza- 
tion discussed  in  this  paper  a  group  whose  primary 
responsibility  is  ensuring  that  company  standards  are 
enforced.  The  Quality  Control/  Quality  Assurance  Branches 
are  -^.he  organiza -^ions  tasked  with  the  job.  It  is  therefore 
the  purpose  of  this  paper  to  examine  the  Quality  Control 
programs  of  various  government  and  aon-government  organiza- 
tions that  produce  software  and  have  established,  well 
documented  standards.  The  effectiveness  of  their  program 
and  the  method  of  operation  within  their  company  structure 
will  also  be  discussed. 
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In  Chap-t'^r  2,  ths  authDrs  will  list  and  identify  currsn- 
trends  and  stats  of  ths  art  processes,  techniques,  and 
methods  that  have  been  cDllated  through  an  examination  of 
current  literature  about  quality  and  the  software  develop- 
ment process.  In  Chap-s::  3  and  Chapter  ^  we  will  discuss 
the  Quality  Control  programs  at  TRW,  General  Electric,  Naval 
Oceans  Systems  Command,  and  the  Fleet  Material  Support 
Office.  Finally,  in  Chapter  5,  a  set  of  recomendations  with 
justification  will  be  provided  for  FMSO  consideration  during 
the  planning  and  execution  of  future  corporate  policy  and 
expansion  with  emphasis  on  the  Quality  Control  effort. 
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II.    QOailTY    ASSgRANCE    rECHNIQOES 

The  rapid  expansion  of  the  computer  industry  in  the  past 
ten  years  has  been  accompanied  by  an  increase  ±n  the  problem 
of  producing  a  quality  software  product.  Surveys  have  shown 
that  as  much  as  60  percent  of  softwara  produced  have  serious 
faults  in  the  first  iteration  -  faults  serious  enough  to 
cause  the  prograi  to  fail  in  its  task  [Ref.  2].  A  tradi- 
tional facet  of  American  manufacturing  has  been  strict 
adherence  ro  quality  staniards  and  -lis  prcduction  of  :^cft- 
ware  should  be  no  different.  Although  computer  software  is 
sometimes  considered  more  of  an  art  than  an  engineered 
product,  computer  professionals  agree  that  software  quality 
assurance  is  an  important  part  of  the  software  life  cycle 
and  most  data  processing  departments  now  coniain  seme  sort 
of  quality  assurance  function.  Many  of  the  significant 
elements  of  production  engineering  such  as  documen-ationr 
testing,  projec-  management  and  quality  assurance  are  now 
emerging  as  an  integral  part  of  a  software  production 
activity.  Such  things  as  structure!  programming,  software 
engineering  and  quality  assurance  are  fast  becoming  the  norm 
rather  than   the    exception.  User   satisfaction,       compliance 

with  approved  methods  of  building  applications,  organiza- 
tional goals,  and  performance  goals  have  all  been  driving 
forces  behind  this  movement.  This  chapter  deals  with 
quality  assurance  in  the  :romputer  industry  and  its  role  in 
software    development. 
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A.       QUALITY    AND    CUALITY    ASSURANCE 

'J  •      Quality    Cef  ir.ed 

Quality      is,      at      best,      a     relative   and      subjective 

measurement.  The      American      Heritage      Dictionary      defines 

quality  as  a  characteristi:::  or  attribute;  a  property.  It 
also  is  thought  of  as  the  natural  or  essential  character  of 
something.  In  everyday  life,  quality  measurements  are  done 
continually  and        usually  without         second  thought. 

Side-by-side  comparisons  of  objects  under  identical  condi- 
tions and  with  predetermined  concepts  form  the  basis  of  most 
comparitive  judgements.  0  nf  or tunat,  =  ly ,  these  decisions  ars 
usually  unique  and  have  little  value  to  anyone  else  unless 
they  are  made  by  an  expert  [ Ref  .  3].  One  widespread  opinion 
is  that,  by  its  very  nature,  quality  defies  definition  and 
must  be  uniquely  defined  for  the  ireii  in  question  by  stating 
a  list     of   characteristics    and   attributes.  This    technique 

implies  no  evaluation  or  judaement  Df  the  item,  but  .iierely 
provides  descriptive  traits  by  which  the  appraiser  may  form 
an  opinion  [Ref-  4].  Ken  Johnson,  5.  sof-ware  quality  assu- 
rance manager  in  industry  and  chairman  of  a  working  party 
set  up  by  the  Electronic  Engineering  Association  concerned 
with  software  quality  assurance,  disagrees  somewhat  with 
other  definitions  concerning  the  ephemeral  attribute  of 
quality  and  declares  that  quality  is  "the  totali-y  of 
features  and  characteristics  of  a  product  or  service  which 
bear  on  its  ability  to  satisfy  a  given  need.  In  short,  it 
is  a  fitness  for  a  purpose  at  an  economic  cost"  [Ref.  2].  A 
recent  study  pointed  to  a  number  of  myths  concerning  quality 
in  organizations.  These  include  suoh  things  as  quality  is 
impossible  to  measure,  quality  lowers  productivity,  quality 
means  poor  workers,  and  quality  is  the  responsibility  of  the 
"quality  department".  It  goes  on  to  give  the  sharpest  defi- 
nition   of   qualit y-" quality      is    the    sum    costs      of    prevention. 
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appraisal  and  failure"  [Ref.  5].  This  definition  sheds 
light  on  the  subject  bat  is  still  not  as  comprehensive  as 
may  b€  desired.  Hence,  expectations  concerning  quality 
measurements  must  be  tempered  with  realistic  Jcaowledge  that 
any  measurement  will  be  partially  imperfect  or  imprecise. 
When  dealing  with  software,  in  itself  not  the  most  tangible 
of  products,  confidence  levels  and  error  tolerances  play  an 
important  role  in  determining  acceptable  qualiry  levels. 

2-   Qiialiil  Assurance  Defined 

Dnlike  the  concept  of  quality,  which  is  -.isually 
Thought  of  as  an  a^Ltribute  of  a  good  or  service,  qualify 
assurance  is  most  often  related  to  =  process  or  methodology. 
According  to  Frank  Ingressia  from  TRW  Corporation's  Defense 
and  Space  Systems  Group  (DSSG),  "quality  assurance  is  a  lot 
like  sex,  freedom  and  democracy.  Everyone  is  for  it,  but 
only  under  certain  conditions."  There  are  a  host  of  defini- 
tions which  apply  to  quality  assurance.  The  official  Air 
Force  position  is  that  ^jality  assurance  is  a  discipline 
which  provides  adequate  assurance  *:hat  material,  data, 
supplies  and  services  oonform  to  established  technical 
requirements  and  achieve  satisfactory  results  [Ref.  6]. 
This  definition  is  much  more  encompassing  than  mcs-,  early 
interpretations  which  called  for  quality  assurance  to  merely 
verify  conformance  to  specifications.  At  the  other  end  of 
the  spectrum  is  the  feeling  that  quality  assurance  is  merely 
the  business  of  ensuring  that  the  product  is  not  the  result 
of  good  luck  but  rather  the  inevitable  reward  for  good 
management  practices.  Still  another  definition  and  perhaps 
one  that  will  become  widely  used  has  been  proposed  by  the 
Institue  of  Electrical  and  Electronic  Engineers  in  their 
reoent  study  concerning  software  quality  assurance  standards 
[Ref.  7]«  It  states  that  "quality  assurance  is  a  planned 
and  systematic  pattern  of  all  actions  necessary   to  provide 
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adequate  confidence  that  the  itera  or  product  conforms  to 
established  technical  regii  irements".  This  definition  has 
been  borrowed  frcm  MIL-STD-1093  and  is  consistent  with  th= 
accepted  usage  of  the  tern.  Conforraing  to  specifications, 
ensuring  good  management  practices,  testing  of  requirements, 
confidence  that  the  system  is  reliable  or  the  product  is 
desirable — all  of  these  things  are  considered  goals  of  a 
quality  assurance  plan.  The  bottom  line  for  a  quality  assu- 
rance function  is  ensuring  that  user's  needs  have  been 
adequately   satisfied.  There   are   a    multitude     of   different 

philosophies  concerning  the  achievenent  of  these  goals  and 
they  will  be  briefly  addressed  in  a  later  sect-ion  of  this 
chapter. 

3.      Traditiq nal    ^ualitv    Assurance    in    Industry 

For  manufactured  products,  quality  usually  means  a 
combination  of  quality  of  design  and  of  manufacture 
[Ref.  8].  Quality  of  design  is  tha  value  inner sn-  in  -he 
design;  a  measure  of  the  excellence  of  rhe  design  in  rela- 
tion to  the  cust-cmer's  requirements.  The  quality  contrcller 
has  the  responsibility  to  ensure  that  the  quality  level 
determined  by  management  can  be  achieved  on  production 
equipment.  Quality  of  manufacture  is  a  measure  of  how  well 
the  product,  at  acceptance,  conforms  to  the  design.  There 
are  most  often  five  basic  stages  of  quality  control  in  a 
factory  [Ref.  8].  These  consist  of:  (1)  Deciding  what  to 
manufacture  and  prepare  specifications  covering  all  require- 
ments, (2)  Make  pre-production  checks  and  work  out 
organizational      responsibilities,         (3)  Production,         (U) 

Feedback  on  quality  deficiencies,  and  (5)  Establish  long- 
term    quality   plans. 

The  quality  control  function  in  a  manufacturing 
plant  is  usually  a  very  complex  and  intricate  organization. 
It    becomes    involved    in  all      phases   of    the    production    process 
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and  serves  as  the  ongoing  checker  for  production  results. 
Tha  role,  structure  and  objectives  of  guali"y  assurance  in 
software  production  will  b2  examined  in  detail  later  in  the 
chapter,  but  there  are  many  similarities  which  exist  and 
indeed     industrial    guality      assuranc?      forms      the    basis      for 
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Figure  2,1    Typical  Quality  Assurance  Organizational  Structure, 

software  quality   assurance  techniques.    Figure   2.1  illus- 
trates a  typical  manufacturing  quality  assurance  department. 
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B.   THE  HOLE  OF  COALITY  ASSURANCE 

The  statistics  concerning  the  growth  of  the  software 
industry  in  the  past  15  years  as  well  as  the  problems 
concerning  this  growth  are  quite  well  documented  and  recog- 
nized. For  example,  one  company,  Boeing  Aerospace,  reported 
that  on  a  large  software  project;  a),  only  1U  percent  of  the 
total  number  of  runs  would  have  been  required  had  there  been 
no  errors  or  failures,  anl  b) .  39  percent  of  the  runs,  while 
successfully  completed,  were  later  invalidated  because  of 
data  errors,  tape  failures,  or  program  bugs  [Ref.  9].  From 
a  cost  standpoint,  over  eight  billion  dollars  was  spent  for 
software  of  various  types  in  1980  [Ref-  10].  Compu-ers,  and 
in  turn  software,  are  becoming  entangled  in  every  aspect  of 
our  daily  lives.  From  electronic  banking  and  shopping  to 
NASA  space  projects  to  more  effecient  use  of  farm  machinery, 
we  rely  en  computer  software  to  help  us  make  more  and  more 
of  our  decisions.  Software  developers  have  recognized  their 
responsibility  towards  quality  software  and  mcs'  data 
processing  departments  no»^  incorporate  some  sort  of  quality 
assurance  function  into  the  production  of  software.  This 
quality  assurance  function  is  able  to  alleviate  the  problem 
most  software  managers  have  of  becoming  involved  in  the 
system  at  a  point  when  the  cost  becomes  significant  and  the 
dates  of  iraplemGntation  approach.  The  establishment  of  a 
quality  assurance  function  provides  management  with  a  degree 
of  confidence  that  an  independent,  technically  trained  group 
is  monitoring  the  goals,  methods  and  performance  of  applica- 
tions from  the  beginning  of  the  project. 
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2.      Ob  jectiv  ^s    of  2ii§.lil2  Assurance 

The  quality  assurance  function,  as  part  of  the 
systems  production  group,  works  to  ensure  that  standards 
concerning      goals,      methods      and   objectives      are    met.  The 

quality  assurance  group  typically  performs  those  functions 
that  the  data  processing  manager  might  do  personally  if  time 
permitted.  Quality  assurance  reviews  each  system  to  ensure 
compliance   wiht    the    following   i-ems.  The   system    must    meet 

the  needs  of  the  user  department  and  other  users  and  at  the 
same  time  not  infringe  on  the  rights  of  other  systems  users. 
The  goals  of  the  system  should  be  consistent  with  the  objec- 
tives of  the  entire  organization.  If  there  is  a  conflict, 
the  goals  of  the  organization  should  maintain  priority  over 
the  goals  of  one  user.  The  system  goals  should  also  mesh 
with  the  EDP  department  objectives  and  if  there  is  a 
conflict,  it  shculd  be  resolved  before  implementation.  If 
there  are  external  industry  or  government  requirements,  the 
goals  of  the  system  should  confDCtn  to  these  standards. 
Controls  on  the  system  must  be  complete  (management 
controls)  and  the  system  must  be  auditable.  The  system 
should  conform  to  all  general  policies,  procedures,  stan- 
dards and  guidelines  established  by  the  organization  and  the 
electronic  data  processing  department.  Quality  assurance 
must  finally  ensure  that  the  design  of  the  system  is  econom- 
ical (least  cost  system),  effective  (desired  results  with 
minimum  effort)  ,  and  efficient  (maximize  use  Df  people  and 
machines) . 

3-      Costs    of   Quality.   Assurance 

The  cost  of  a  quality  assurance  function  is  very 
difficult  to  estimate  or  even  measure.  William  Perry,  an 
author  of  extensive  material  on  software  quality  assurance 
and    a   member   of    the      Quality   Assurance    Institute    in   Orlando, 


Florida,  cones  closest  to  a  precisa  figure  by  saying  that 
"if  qualiiiy  assurance  is  included  as  a  line  i-em  in  a 
project's  budget,  it  should  range  sDiiiewhere  between  2.5  and 
5  percent  of  the  total  project  cost"  [  Ref .  4].  Figure  2.2 
is     indicative  of     the     percentage   of      costs      expended   on      a 
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Figure   2.2        Quality    Assurance    Project  Costs. 

typical  five-phase  software  development  project.  This  esti- 
mate is  still  rather  idealistic  because  of  the  differences 
which  may  arise  because  of  staffing  alternatives,  metho- 
dology, lifecycle  entry,  the  difficulty  in  defining  quality 
assurance   and   other    unknowns. 

Cost  justification  is  an  important  aspect  of  imple- 
menting a  quality  assurance  function.  In  the  hardware 
arena,  the  cost  of  quality  assurance  is  most  often  justified 
by  lower  warranty  cost  markups  in  f^e  price  of  the  product. 
This  is  equally  true  in  the  software  world.  Warranty  costs 
will  be  lower  for  a  quality  software  product.  Another 
fringe  benefit  of  a  quality  software  product  is  its  ease  of 
adaptability   to    a  similar  product  at   a    lower   cost    [Ref-    11]- 
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Ths  cos-  savings  of  an  efficient  quality  assurance  ac-ivi-y 
is  often  hard  to  justify  because  of  the  indiscipline  of 
those  not  qual  ity-consciD  us  often  makes  measurement  of 
improvements  very  difficult.  As  is  usually  the  case, 
results  speak  the  loudest.  The  cost  savings  involved  in 
having  projects  done  on  time  and  within  budget  allows 
quality  assurance  to  maintain  its  Isvel  of  efficiency  and 
reduces  management  time  spent  sorting  out  the  mess  which 
results  from  bad  planning. 

Finally,  it  has  been  shown  that  the  entry  point  of 
the  quality  assurance  function  into  the  lifecycle  of  the 
pro  jeer  has  a  definite  affect  or.  tha  cost  [Hsf.  12].  The 
scope  and  structure  of  the  quality  assurance  effort  is 
affected  strongly  by  the  cost  of  errors  related  to  the  phase 
of  development-  Figure  2.3  is  a  typical  illustration  of  the 
economics  of  error  detection  in  th=  various  phases  of  devel- 
opient.  As  -his  figure  shows,  the  earlier  a  problem  is 
detected,  the  less  expensive  is  th=  cost  of  correction. 

C.   SOFTWARE  DEVELOPMENT 

1 •   Software  Lifecycle  Phase s 

Referring  to  the  lifecycle  of  software  is  the  most 
common  mexhod  of  addressina  the  development  of  a  software 
product.  A  review  of  the  liturature  has  produced  a  plethora 
of  illustrations  of  what  is  considered  to  be  the  true  or. 
ideal  lifecycle.  Most  examples  use  different  terminology  to 
describe  what  is  happening  at  a  particular  stage  in  the 
lifecycle,  but  they  all  tend  to  include  the  critical  items. 
Tha  simplest  lifecycle  found  is  one  in  which  there  are  only 
three  stages  -  design  and  development,  active  and  passive 
[Ref.  13].  Conversely,  the  most  complex  definition  of  a 
software  lifecycle  contains  eight  phases-systems  definition, 
software   allocation,    specification,     design,     code. 
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Figure   2. 3        Belative   Error  Correction   Costs. 

verification,  integration,  and  operation  [ Ref .  14],  Table  1 
depicts  a  comparison  of  tha  various  phases  of  the  lifecycle 
of   software. 

Whatever  the  phases  of  software  development  are 
called,  there  are  certain  items  which  must  be  accomplished. 
The  beginning  of  a  projact  may  be  designated  as  a  feasi- 
bility study,  reguirements  definition,  systems  definition, 
user  requirements  study,  initiation  study  or  something  else, 
but  it  consists  of  all  activities  which  deal  with  deter- 
mining whether  or  not  a  software  project  should  be 
initiated.  Such  things  as  cosz-bsnaf it  studies,  goal  defi- 
nitions, and  documentation  requirements  are  typical 
activities    which      should   be    accomplished    here.  Next    comes 
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TABLE    I 
Software   Lifecyclo   Coaparison 


AUTHOR  LIFSCYCLE    PHASES 

PERRY    Feasibility  -    Design   -    Programming    -    Testing 

Conversi  on 

FUJII    Conceptual    Design    -   Requirements   Definition   - 

Design   -    Code    and   Checkout   -    Testing   - 
Integration  -   Operational 

MENDIS Design   -   Zola    and    Debug    -    Qualifi::at  ion    Test 

ROBERTS Design    and    Development    -    Active   Stage   - 

Passiv5    Stage 

HOWLEY 1-   Systems    Definition   -    Software   Allocation   - 

S ':!ecif  13  a"^ions   -    D^sian    -    Code    - 
Verification    -    In'iegration   -    Opera-ion 

DUNN  and 

ULLMAN Us<=r  Requirements  -  Svstsm  Functional  Specs  - 

Software  Functiohil  Specs  - 
Impl5ra9nt.at.ion  -  Verification  and  Tes-  - 
Operations  and  Maintenance  - 
Configuration  Managemen-*- 

FTPS  PUB  38  -  Initiation  -  Devslopmeit  -  Operation 


the  general  design  of  the  system.  This  stage  is  normally 
labeled  design,  design  and  development,  software  design,  or 
systems  functional  specifications.  Activities  such  as 
design  alternatives,  specific  requirements,  functions  to  be 
performed,  and  program  and  data  base  specifications  should 
be  included  in  this  phase.  The  next  phase  is  probably  the 
mosx  rudimentary  in  terms  Df  work  and  deals  with  programming 
and  testing.  This  stage  is  also  referred  to  as  coding  and 
debugging,  verification  and  validation  (after  coding  is 
complete),  or  is  sometimes  broken  into  two  distinct  phases. 
The  system  is  now  written  in  the  desired  language  and 
various  tests  are  performed  to  ensure  the  system  performs  as 
desired.  The  next,  and  most  often  last  phase,  is  referred 
to  as  conversion,  integration,  operations,  implementation, 
maintenance,    or  configuration   [na:iagement.     This   stage 
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consists  of  laaintaining  "-he  software,  performing  ongoing 
evaluations  and  changing  it  as  addit.ional  requirements  ars 
identified. 

2.      Software    Pro2erti9s 

As  stated  previously,  software  is  an  elusive  product 
upon  which  to  place  a  quality  oieasuremsnt.  Howley  and  Fink, 
software  quality  engineers  for  the  Boeing  Aerospace 
Corporation,  have  attempted  to  verbalize  what  a  quality 
software  product  is  by  stating  "A  quality  software  product 
may  be  defined  as  one  whi^h  exhibits  the  following  proper- 
ties: it  satisfies  the  software  specif icaticr.  and  d-sign 
requirements;  it  performs  all  intended  functions;  it  is 
relatively  free  of  design,  interface  and  coding  deficien- 
cies; it  has  a  low  life-cycle  oost;  it  is  properly 
identified  and  documented;  and  it  incorporates  all  needed 
software  quality  characteristics"  [Ref.  15].  To  achieve  the 
level  of  quality  which  is  desired  by  the  above  defini'^ion, 
there  are  a  number  of"  factors  which  contribute  to  the 
production   of      quality  software   [Ref.     16],  These    include, 

but    are   not    limited    to: 

1.  Correctness  -  This  generally  leans  programs  perform 
in  exactly  the  manner  specified  in  the  program  docu- 
mentation. Correctness  is  usually  considered  an 
ideal  quality    which    is   rarely    achievable. 

2.  Reliability  -  This  attribute  means  that  programs 
perform  relatively  trouble  free  all  the  functions 
expected    from    the    specifications    or    documentation. 

3.  Validity  -  Validity  is  concerned  with  the  question  of 
whether  the  functions  and  performance  of  the  programs 
are  adequate  and  suitable  to  a  needed  purpose.  The 
software,  without  manual  intervention  or  additional 
programming,  should  perform  the  functions  that 
reasonably  would   be    expected      of    it.         This    attribute 
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is  a  very  subjsctive  one  aad  must  be  flexibis  t^ 
changing  lequiremeQi:  s. 
U.  P.ssilience  —  This  ine2.r.s  t.h?."'!!  prcgr'snis  shcild  be 
designed  in  such  a  way  to  be  forgiving  of  common  user 
and  data  errors.  Inconsistent  or  unacceptable  data 
entries  shouldn't  provoke  actions  which  make  no  sense 
to   the  user. 

5.  Usability  -  Human  factors  and  limitations  and  conve- 
nient usage  techniques  should  be  considered  whenever 
a    program   is    written. 

6.  Clarity  -  Programs  should  be  easily  understandable 
from  the  users  manual  and  all  -he  documen-ation 
should  be  clear,  concise  and  rogent.  Programs  should 
be  modularly  designed,  h=v?  explanatory  ccmmsn-s 
where  necessary,  and  use  mea:iingful  choices  of  vari- 
able names. 

7.  Maintainability  -  3ood  documentation  and  comments  as 
well  as  clear  structure  will  make  programs  more 
easily  repairable.  Clariry  is  also  essential  for 
making   minor    improvements. 

8.  iJlodifiabilit y  -  Major  changes  should  be  anticipated 
and  the  software  designed  so  that  program  functions 
that  might  require  major  change  are  well  documented 
and    isolated    in   distinct    modules. 

9.  Generality  -  Programs  shouli  be  applicable  to  a  wide 
range   of    input   values   and   usage    modes. 

10.  Portability  -  Programs  should  be  easily  adaptable  to 
transfer  to  another  computer  system  or  operating 
system. 

11.  Testability  -  Programs  should  be  simply  structured 
and  use  g^neraly  algorithms,  to  facilitate  step-by- 
step   testing    of  all    capabilities. 

12.  2fficiency  -  The  attempt  should  be  made  to  keep  the 
cost   of    program  operation    as    low    as    possible. 
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The  preceding  list  contains  many  of  the  ittribatas 
of  a  quality  software  product  but  is  not  the  only  list  avai- 
laDle.  Other    authors      include   such      things   as      integrity, 

flexibility,  reusability,  interoperability,  and  others  which 
are    descriptors    of    quality    software.  Figure   2.4    is   a    good 

illustration  of  how  these  factors  affect  each  other  and  what 
degree  of  a  certain  factor  is  required  when  a  different 
factor  is  recognized.  As  can  be  seen,  some  factors  are 
synergistic        while      others        conflict.  The      impact        of 

conflicting  factors  is  that  the  cost  to  implement  will 
increase.  This  will  serve  to  lower  benefit  to  cost  ratios 
[Ref.    17]. 

3.      Hardware  Characteristics  vs.  Software 

Characteristics 

Hardware  is  a  tangible  piece  of  engineering.  It  has 
very  precise  specifications  and  drawings  and  is  based  on 
well  established  building  principles,  the  aim  being  to  manu- 
facture many  identical  (or  near  identical)  items.  The 
design-development- prcduc-^.i  on  cycle  is  mature  and  well 
tuned.  Refinements  to  the  design  may  be  made  many  times 
before  a  commit  irent  to  manufacture  is  made.  In  contrast, 
software  engineers  ship  their  prototypes.  Software  is  a 
largely  intangible  product,  only  dascribed  by  many  volumes 
of  specifications  and  listings.  Software  is  unlikely  to  go 
through  as  many  prototype  stages  and,  therefore,  the  oppor- 
tunity for  design  iteration  and  Improvement  is  somewhat 
limited   [Ref.    18  ]. 

Most  aspects  of  hardware  are  functionally  testable 
and  have  very  specific  requirements  testing  programs.  It  is 
fenced-in  by  established  principles  aad  well-known,  widely- 
used  disciplines.  Unlike  hardware,  software  is  functionally 
non-  testable  in  all  but  the  simplest  of  computer  programs 
and    as      a   result,        it   is      very    difficult      to   tes-      software 
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Figure   2,4        Relationships    Between   Software   Quality   Factors. 

completely  [Ref-  19].  Figure  2.5  illustrates  -^he  problem  of 
testing  software.  Each  circle  represents  a  processing  node. 
The  clockwise  arcs  are  jumps  around  the  individual  nodes, 
and  the  counter-clockwise  arcs  specify  the  number  of  itera- 
tions of  each  half.  The  number  of  discrete  states  possible 
within  this  trivial  diagram  is  approximately  100  quadrillion. 
If  these  could  be  tested  at  the  staggering  rate  of  ons 
per    microsecond,       it    would    have      bsen    necessary    to    start    the 
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Figure    2.5        Computational   Paths. 

t9s-ing  ever  2,000  years  ^go  to  !!^  =  =*-  r.ex-''  month's  scheduled 
delivery. 

The  usual  cause  of  hardwar=  failure  is  component 
deterioration.  Sof-ware  failures  are  almost  always  design 
errors  thar  show  up  only  when  the  software  is  used  under 
certain  conditions.  Hence,  Quality  Assurance  techniques  for 
software    focus  on  getting   the   design    right   [Ref.    20]. 

A  comparison  of  hardware  and  software  lifecycles  is 
offered  by  Table  2  and  shows  clearly  that,  similar  terms  in 
the    two    fields    sometimes    have   radically    different    meanings. 

^ •      Software   Ma nagement 

The  driving  forces  behind  implementing  managemenr: 
s-cructure  in  any  organization  are  reduction  of  costs, 
increased  control,  and  producT.ion  of  a  quality  product. 
Software  production  is  no  different.  Figure  2.6  depicts  the 
rise    in      software  costs      in    the    last      twenty    years.  As    is 

apparent,  software  costs  greatly  exceed  equipment  costs  over 
the  useful  life  of  computer  services.  With  this  kind  of 
growth  it  is  imperative  that  software  be  managed  to  minimize 
costs.  While  cost  minimization  is  an  important  aspect  of 
software  management,  there  are  other  reasons  of  equal  impor- 
tance. Most   software      in    production      today,     is    a      complex 
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TABLE  II 
Hardware  and  Software  Product  Life  Cycles 


DEVELOPMENT 


HARDWARE 

Determine   user 

Reguirement  s 
Devej-op  Product 

Concept  (Fun  ctio  nal) 
Specify  Component 

Design    (Detailed) 
Build    and   Test 

Prototype 
Develop  Manufacturing 

Techniques 


SOFTWARE 

Determine  user 

Reauirements 
Develop  Product 

Concept  (Functional) 
Specify  Compcn<=nt 

Design  (Detailed) 
Implement  and  Test 

Programs 


I    Manufacture   Product 
INSTALLATION)    Make    Producr    Available 

to    Users 


Copy    Programs 
MaKs    Program 


Available    to    Users 


MAINTENANCE- 
IMPROVEMENT 


Maintenance    (Correct 
Component   Failures) 

Recall   Product  to 

Correct    Design   Flaws 

Enhance   Product 


Maintenance (Correct 
Implementation   and 
Desian    Errors) 

Maintenance (Adaated 
to    Char. a e a 


User 


Environmen-) 


I    Unit    is   Unusable   and 
PHASE-OUT         1         Unrepairabi e(Replacs) 
Prcduc-   is   Obsolete 


ProducT    is    Obsolete 


technical  activity  that  must  be  directed  effectively.  The 
complexity  of  any  program  in  all  but  the  simplest  of  appli- 
cations is  such  that  programming  has  evolved  past  a  routine 
effort  that  can  go  unsupervised  or  be  done  by  junior 
personnel.  Any  investment  of  funds  or  resources  is  likely 
to  be  a  major  one  for  any  organization  and  the  technical 
choices  may  have  widespread  effects  throughout  the  organiza- 
tion. Management  and  technical  control  by  professionals  is 
essential  for  resolving  design  issues  and  giving  adequate 
direction      to         programmers      regarding      cost        and      schedule 
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Figure   2.6        Estimatad   Growth    Df    Software   Costs. 

accountability  of  project  decisions  and  objectives  and  gives 
top  management  visible  maa suras  of  success  in  the  accom- 
plishment of  goals.  As  a  general  rule,  software  projects 
are  often  ini-^.  iated  by  management  personnel  who  control 
budgets  and  schedules  and  software  designers  freguantly  end 
up  doing  a  substantial  amount  of  independent  work  with 
little  pressure  to  evaluate  their  progress  or  the  remaining 
work      or    costs.  With   a      scftwars      managment    structure      in 

place,  important  issues  and  objectives  such  as  cost,  quality 
and  schedule  can  be  carefully  evaluated  and  the  appropriate 
responsibility      for      the        decisions      assigned.  Technical 

controls,  working  procedures  and  rasource  management  are 
further  justifications  of  a  strong  software  management 
structure.  Questions      about   systen      feasibility,         system 

quality,  design  methodology  and  testing  procedures  are  ones 
which   should     be   answered   by     software    management.  All    of 

these  things  help  designers  and  programmers  to  organize  and 
direct  their     efforts   efficiently      in    solving      such    problems 
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within  available  cost  and  t  ima  limits.  Jlnother  reason  for 
software  mar.agement  is  the  ongoing  problem  of  main-.enance 
after      i  apleaent  ation.  Some      estisiates    put      the      cost      of 

maintenance  at  7  0%  of  total  software  costs.  It  is  important 
to  rGCognize  that  maintenance  is  just  as  important  as  devel- 
opment. Good  software  management  principles  will  attempt  to 
be  ongoing  throughout  the  lifecycle  and  will  make  the 
strongest  effort  to  stress  close  management  of  effort  toward 
the    most    needed    software    capabilities. 

The  management  of  software  development  is  often 
referred  to  as  "software  engineering".  This  implies  -hat 
the  principles  cf  produc-ion  engineering  mandgement  can  be 
transferred  or  applied  to  software  development.  Software 
engineering  suggests  that  "the  entire  development  of  a  soft- 
ware product  from  initial  conception  through  design, 
implementation,  testing,  and  maintenance  can  be  organized  in 
a  systematic  and  manageable  fashion.  It  should,  therefore, 
be  possible  to  monitor  the  quality,  performance  and  cost  of 
the  end  product  through  the  several  phases  of  its  life 
cycle"    [Ref .    21 ], 

5.      Standards  for   Software   DevelDEIEilll 

An  area  which  has  been  seriously  neglected  in  the 
software  development  industry  during  its  growth  has  been  the 
establishment  of  standards  cf  conformance.  There  has  been  a 
recognition  of  this  lack  during  the  past  few  years  and 
attempts  have  been  made  to  provide  adequate  standards.  The 
most  widely  used  military  document  concerning  standardiza- 
tion of  quality  assurance  plans  i=  MIL-3-52779A  dated  1 
August  1979.  This  document  is  applicable  to  Department  of 
Defense  agencies  when  acqairing  software  where  the  acquisi- 
tion involves  either  software  alone  or  software  as  a  portion 
of   a      system   or    subsystem.  It    provides      specific   guidance 

concerning   software      quality  assurance      program    requirements 
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and  covers  such  •things  =s  tools,  tichniq^ies  md  msthodclc- 
giss,  ccmpu-^er  program  design,  work  car tif ication,  documen- 
tation, computer  prcgratn  library  controls,  reviews  and 
audits,    configuration   management   and   testing  [Ref.    22], 

This  document  is  ased  not  only  by  DOD,  but  has  also 
been  referred  to  by  many  civilian  organizations  in  the 
absence  of  anything  better.  The  Institute  of  Electrical  and 
Electronic  Engineers  has  recently  sponsored  a  committee  to 
evaluate  the  problem  and  develop  a  set  of  software  quality 
assurance  standards.  Their  stated  purpose  was  to  "provide 
uniform,  minimum  acceptable  requirements  for  the  preparation 
and  content  of  sof-ware  quality  assurance  plans"  [aef.  23]. 
The  sections  of  the  standard  developed  contains  direction 
concerning  such  -^hings  as  reference  documents,  management, 
documentation,  standards,  practices,  and  conventions, 
reviews  and  audits,  configuration  management,  problem 
reporting  and  corrective  action,  tools,  techniques,  and 
met hcdologiss ,  code  control,  media  control,  and  supplier 
control.  As  can  be  seen,  it  is  extensive  and  comprehensive 
in  scope  and  provides  guidance  for  d  =  veloprr.ent  of  a  thorough 
software    quality    assurance    plan. 

The  preceding  two  documents  are  the  most  widely  used 
standards  referred  to  when  developing  a  software  quality 
assurance  plan.  There  are  some  other  publications  which  can 
be  reviewed  for  direction  concerning  development  of  soft- 
ware. Federal  Information  Processing  Standards  Publication 
38  (FTPS  PUB  38)  provides  thorough  and  comprehensive  guide- 
lines for  documentation  of  computer  program  and  automated 
data      systems.  National      Bureau      of        standards      Special 

Publication  500-11  is  a  good  overall  guideline  to  computer 
software    management   and   quality    control.  There    are    a   host 

of  -journal  and  proceedings  articles  dealing  with  software 
quality  assurance  which  provide  information.  All  of  these 
are      not    "official"      guidelines    and      lack    any      authoritative 
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endorsement.  The  best  hopr  for  bet-ter  results  from  sof-wara 
quality  assurance  prcgrams  lies  in  an  accsptancs  of  z'a  I"EE 
Standards        and  conformance        to  their        requiramen-s. 

Unfortunately,  these  are  relatively  new  and  there  is  insuf- 
ficient data  to  declare  that  new  in'^ustry  standards  have 
been    developed. 

D.       SOFTWARE    QUALITY    ASSOSANCE    METHODOLOGY 

1  •      ^li§.liil    Assurance  P lann ing 

Planning  is  essential  for  tha  successful  achiavement 
cf  any  project  and  must  raiain  dynanic  to  be  of  any  use,  I*- 
is  important  that  plans  are  modified  to  reflect  changes  in 
requirements  as  they  occur.  On  a  bread  scopa,  a  quality 
assurance  plan  must  indicate  -he  particular  activities  which 
will  enable  the  required  la  vel  of  quality  to  be  achiaved  on 
any  given  project.  How  the  product  is  to  be  assured  and 
what  acTivi-^ies  the  quality  assuranca  group  is  to  undertake 
in  order  to  satisfy  organizational  requirements  are  key 
elaraents. 

Quality  assuranca  generally  parallels  the  systems 
development    process.  Tha    position      of    review      points    will 

depend  upon  management's  requiramants  concerning  decision 
points   or   information  requirements.  The    importance   of   tha 

system  to  the  organizatioQ  as  a  whole  will  determine  tha 
amount  of  time  spent  on  aach  project.  The  critical- points 
are  the  end  of  each  systams  development  life  cycle  phase. 
At  this  time,  an  opinion  is  renderad  by  quality  assurance  as 
to  the  adequacy  of  the  dasign  process  up  to  that  point. 
This  opinion  can  then  be  jsed  as  a  decision  making  factor  in 
determining      whether   to     progress   to      the      next    phase.  In 

discussing  the  review  criteria  a  five  phase  systems 
lifecycle   will   be  assumed. 
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The  feasibili"ty  study  cominonly  consis'^-s  of  =vilua- 
■^ion  of  alternatives  and  techniques  -o  solve  a  particular 
problem  and  reccmmend  a  course  of  action  to  aenag^'insnt. 
This  may  or  may  not  include  a  computer  system  and  the 
personnel  involved  in  the  study  may  or  may  not  have  computer 
experience.  The  role  of  quality  assurance  during  the  feasi- 
bility study  is  usually  one  of  a  consultant  to  discuss  the 
practicality  of  alternatives  or  cost  estimates.  At  the  end 
of  the  feasibility  s-udy,  quality  assurance  should  evaluate 
whether  the  study  team  followed  the  organization's  proce- 
dures in  developing  a  proposal  for  management  and  comment  on 
any  computerization  aspects  of  the  proposal. 

The  next  phase,  design,  is  critical  to  guality  assu- 
rance. The  greatest  impact  is  made  during  this  phase  and 
the  quality  assurance  group  should  strive  to  impact  the 
design  without  actually  participating  in  the  design  process, 
Tha  goal  is  to  not  argue  for  or  against  particular  designs 
but  to  review  the  proposed  design  on  merit.  One  method 
which  is  widely  used  in  this  phase  is  to  divide  the  design 
into  two  phases -informal  and  foraial.  The  informal  phase 
occurs  after  a  preliminary  design  is  on  paper  and  consists 
of  quality  assurance  giving  discussion  only  review  to  allow 
the  design  group  to  determine  if  they  are  on  the  right 
track.  There  is  no  report  to  management  generated  from  this 
phase.  A  structure  such  as  this  requires  a  good  working 
relationship  between  quality  assurance  and  the  design  group. 
At  the  end  of  the  design  phase,  a  formal  review  occurs  and 
the  design  is  normally  fixed  at  this  point.  Compliance  to 
performance  criteria,  systems  goals  and  procedures  are 
reviewed  by  quality  assurance. 

Program  design  and  program  coding  make  up  the  next 
phase  which  must  be  evaluated  by  quality  assurance.  While 
these  two  activities  may  be  combined  into  one  phase,  it  is 
usually   more   effective   and   it  facillitates   structured 
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proqramraing  ro  separate  them.  3y  reviewing  ths  prcaram 
design,  quality  assuranca  has  a  greater  opportunity  to 
ensure  compliance  to  design  proceduras  and  standards.  This 
will  hopefully  alleviate  many  problems  usually  encountered 
in  the  coding  phase.  At  the  end  of  the  programming  phase, 
quality  assurance  should  perform  a  review  to  ensure  compli- 
ance to  procedures  and  standards  for  such  things  as  coding 
and  use  of  operating  system  facilities.  This  should  be  a 
detailed  review  and  guali-^y  assurance  must  examine  all 
aspects  of  coding,  operating  system  instructions,  file 
structures  and  anything  else  which  will  affect  the  operation 
of  the  computer. 

In  the  n  «=xt  phase,  system  testing,  quality  assurance 
is  mostly  concerned  that  an  adequate  test  plan  has  been 
prepared,  that  it  is  followed  and  that  it  conforms  to  the 
standards  of  the  organization.  Quality  assurance  will  only 
review  test  results  to  ensure  compliance  to  standards  and 
should  not  become  involved  in  a  detailed  test  plan.  At  the 
end  of  the  system  testing  quality  assurance  should  again 
review  for  conformance  to  organizational  policy  and  check 
for  user  satisfaction. 

The  last  phase  in  our  example,  implementation,  can 
be  the  broadest  in  scopa  and  longest  in  duration.  This 
phase  is  sometimes  called  conversion  and  is  generally 
thought  of  as  the  process  of  replaceaent  or  new  installment. 
As  with  testing,  the  primary  concern  of  quality  assurance  is 
that  a  bonafide  plan  has  been  defined  and  that  it  is  beinq 
followed.  The  completion  of  the  implementation  phase  brings 
the  final  review  by  quality  assurance  that  the  procedures 
defined  in  the  design  stage  were  followed.  Once  again,  user 
satisfaction  is  of  paramount  importance  and  quality  assu- 
rance is  reviewing  plans  and  procedures. 
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The  foregoing  example  of  a  quality  assurance  olarx 
over  the  lifecycle  of  a  software  product  is  by  no  means  the 
only  one  to  be  used.  Another  typical  example  is  discussed 
by  Marilyn  Fujii,  a  software  quality  assurance  professional 
from  Logicon,  Inc.,  in  which  the  lifecycle  is  divided  into 
seven  parts  and  quality  assuraace  again  has  a  role  over  ths 
entire  lifecycle  [Ref.  2U]«  The  early  stages  of  the  plan 
are  centered  around  defining  the  procedures  and  standards 
which  will  be  applicable  to  supporting  configuration  manage- 
ment and  computer  program  development.  Most  other 
activities  consist  of  reviewing  and  auditing  software 
products  against,  previously  set  standards,  2uaiit.y  assu- 
rance is  responsible  for  all  design  reviews  and  audits  and 
they  evaluate  all  documentation  such  as  test  plan,  specifi- 
cations, and  users  manuals.  Any  walkthroughs  or  acceptances 
testing  will  be  scheduled  and  conducted  by  quality  assu- 
rance. At  the  delivery  point  in  ths  lifecycle,  they  audit 
tha  final  configuration  to  be  installed  in  -he  operational 
environment.  Figure  2.7  offars  a  visual  presentation  of 
quality  assurance's  role  in  this  example. 

The  examples  given  are  but  two  of  a  multitude  which 
can  be  found  in  professional  literature.  Regardless  of  what 
specific  method  is  used,  there  are  a  number  of  components 
which  should  be  included  in  any  software  quality  assurance 
plan.  The  plan  should  identify  procedures  to  be  used  in 
issuing  work  tasking  instructions  for  all  work  relating  to 
software  development.  Monitoring  of  procedures  and  assuring 
adherence  to  them  should  be  oart  of  any  plan. 
Identification  of  schedules  and  resources  and  tracking 
progress  toward  them  should  be  included.  Work  descriptions, 
responsibility  assignments,  initiation  procedures,  report 
generation  procedures,  and  scheduled  completion  dates  should 
be  addressed.  The  plan  should  document  quality  assurance 
involvement   in  the   specified  development  program.     Also 
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Figure    2,7        Quality  Assurance    Activities. 

provided  should  be  visible  schedules,  niilestones  and 
interdepartmental  dependencies  and  commi-ments.  Levels  of 
detail  required  should  be  included  in  any  plan.  Finally, 
the  organization  responsible  for  software  quality  assurance 
should  prepare  the  plan.  A  general  outline  of  a  typical 
quality   assurance   plan   is    provided   as    Figure    2.3, 

2«      Staffing  and   Organization 

The   success    of  any    qualit.y    assurance    function    begins 
with    the    personn  ^=1      assigned  to    the    staff.  Individuals   as 

knowledgable  as  senior  systems  analysts  and  designers  should 
ideally  be  assigned  to  quality  assurance.  It  is  not  enough, 
however,       for   guality  assurance      personnel    to    merely    possess 


36 


Forward 

Introduc 

•t  ion 

Con* 

e  nts 

Tnts 

n  t 

Product    Description                                                                      1 

1.0 

Scop 

p 

1.  1 

A  pplicabilit y 

1.2 

Ccntracturai    Intent 

1.3 

Relation   to    Other   Contract    Requirements 

2.0 

AP^I 

i  cable   Do cu Bents 

Aiendments   and   Revisions 

3.0 

Requ 
3.  1 

lirements 

Software   Quali-y    Assurance    Plan 

3.1.1    Object  ives                                                                     I 

3.1.  2    Goals                                                                                1 

3.2 

Software   Quality    Assurance    Requirements             | 

3.2.1  Work    Tasking   and    Authorizations                  j 

3.2.2  Configuration    :iauageiii?n- 
3.2.  3    Testing 

3.2.  U    Corrective    Action 

3.2.5    Library   Controls 

3.2.6    Computer  Prograai    Design 

3.2.7    Sof-ware   Documaa ta-ion 

3.2.8    Review    and   Audits 

3.2.9    Tools,    Techniques,    and    M-ehodologies 

3.3 

Subcontractor    Control 

U.O 

Program    Dependencies                                                                    1 

5.0 

Prog 

ram    Risk    Areas                                                                         I 

1 
1 
1 

Figure  2.8   Typical  Software  Quality  Plan  Outline. 

the  characteristics  of  a  good  systems  analyst.  Quality 
assurance  personnel  must  command  the  respect  of  borh  the 
individuals  whose  sysrems  are  being  evaluated  and  the 
management  of  the  SDP  department,  who  musr  rely  on  them. 
The  ability  to  review  the  work  of  others  and  t.o  convince 
them  there  are  tetter  methods  to  perform  their  work  takes 
some  unique  skills  in  dealing  with  people.  Quality  assu- 
rance reviewers  must  have  a  talent  for  good  communicative 
and  persuasive  ability  as  as  well  as  be  respected  for  their 
technical  ability. 
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There  are  a  number  of  variables  concerning  the 
experience  and  number  of  people  assigned  to  quali-^y  assu- 
rance. The  size  cf  the  project  at  hand  is  a  major 
determining  factor.  Small  projects  which  appear  to  be  rela- 
tively simple  and  perhaps  repetitive  of  previous  jobs  may  be 
short-changed  in  the  guality  assurance  area.  If  the  company 
itself  is  small,  it  may  not  be  abls  to  afford  the  commitment 
to  guality  assurance  of  a  large  corporation.  Top  management 
may  not  recognize  the  need  for  quality  assurance  and  hence 
give  it  less  than  prominent  attention. 

The  organization  Df  a  quality  assurance  department 
can  be  set  up  in  maiiy  ways.  One  wiii:ly  held  thaory  is  tha- 
the  higher  in  the  data  processing  structure  the  quality 
assurance  function  reports,  the  better  the  probability  of 
success.  Also,  the  level  of  reporting  is  sometimes  indica- 
tive of  management  support  [ Ref .  4],  Figure  2.9  shows  quite 
a  simplified  view  of  a  rs presentati V9  EDP  department  with 
the  guality  assurance  function  placsd  as  a  staff  function 
reporting  to  the  EDP  Manacrsr.  This  structure  insures  that 
quality  assurance  will  receive  ths  attention  of  the  ZD? 
Manager  and  that  it  will  be  independent  of  all  other  aspeczs 
of  the  data  processing  department.  Figure  2.10  shows  an 
organizational  structure  from  industry^  Informatics  Inc. 
Quality  Assurance  in  this  organization  is  embedded  in  the 
organizational  structure  and  has  secondary  affects 
throughout  the  company.  As  another  example.  Figure  2.11 
shows  a  variation  of  the  placement  of  the  quality  assurance 
function  [Ref.  25].  This  structure,  with  quality  assurance 
as  an  independent  function  reporting  directly  to  a  division 
general  manager,  provides  good  independent  oversight. 

Different  projects  within  an  organization  will 
receive  different  emphasis  in  the  quality  assurance  area. 
The  same  holds  true  for  industry  as  a  whole.  Embedded  soft- 
ware in   a  weapon  system  will   not  receive  the  same   sort  of 
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Figure   2.9        Sample   3DP   Departient   Organization. 

quality  assuranc*?  at-entiDn  as  a  payroll  prcject.  As  each 
projec-  has  differing  requirements,  so  will  each  quality 
assurance      scenario      be  different.  Throughout      literature 

concerning  quality  assurance  i mplemsnt at  ion,  there  are  a 
number  of  mexhods  used  to  organize  a  quality  assurance  func- 
tion. These  have  been  consolidated  into  four  general 
methods  and  will  be  described  in  sorne  detail.  It  iius*:  be 
reaembered,  however,  that  the  type  of  organization  of  the 
quality  assurance  function  will  depend  greatly  on  such 
factors  as  the  type  of  product  and  in.  iustry,  emphasis  given 
quality  assurance  within  the  organization  and  the  scope  of 
the  project.  Quality  assurance  departments  from  selected 
companies  will  be  examined  in  detail  further  in  the  paper 
which  will  expand  en  the  methods  of  organizing  the  quality 
assurance   function. 

The  first  method,  and  probably  most  widely  used  when 
organizations  are  beginning  the  quality  assurance  function, 
is  the  task  force  method.  This  method  allows  organizations 
to  become  involved  in  some  sort  of  quality  assurance 
activity  prior  to  the  formalization  of  the  quality  assurance 
function.       A    task   force   offers   the   advantage   of    developing   a 
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Figure   2,10        Informatics   Inc.    Drganization   Chart. 

group  able  to  handle  th9  unique  problems  which  may  be 
encountered  in  a  given  software  project.  Task  force  members 
with  the  appropriate  background  can  be  hand-picked  by  the 
EDP  manager.  Another  beasfit  is  the  training  afforded  the 
systems  designers  assigned  to  the  taam  because  it  puts  them 
in  a  position  of  analyzing  the  competency  of  systems  design. 
A  disadvantage  to  this  method  is  ths  problem  of  continuity. 
Each  task  force  will  tend  to  develop  its  own  methods  and 
procedures  for  the  review.  If  the  task  fores  members  are 
not  relieved  of  a  significant  amount  of  the  burden  of  their 
daily  work,  then  another  possible  problem  is  that  they  may 
have    trouble    finding    adequate  time   to    devote   to    the   review. 
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Figure    2.11        Traditional  Organizational   Structure. 

A  second  method  is  the  formation  of  a  full  time 
quality  assuranc€  staff.  This  me-hod  prcvid-^s  the  grea-est 
amount   cf   continuity    among    reviewers.  The   ED?    manager   can 

thus  have  a  greater  dsgrae  of  confidence  in  the  quality 
assurance  function.  By  assigning  a  full  -ime  staff,  manage- 
ment is  giving  a  signal  of  the  measure  of  importance  it 
places  on  quality  assurancre.  The  biggest  disadvantage  cf  a 
full  time  staff  is  the  competency  of  the  review  group. 
Whareas  a  task  force  can  add  specialized  knowledge,  a  full 
time  staff  operates  with  the  personnel  assigned.  Another 
problem  is  the  technical  proficiency  of  the  staff. 
Technical  proficiency  with  current  practice  is  very  impor- 
tant both  from  the  standpoint  of  creiibility  of  the  staff 
with  the  rest  of  the  organization  and  the  proficiency  with 
which    the    person  r.el    can   perform    their    function. 

The  permanent  committee  method  is  another  approach 
and  is  basically  just  a  step  up  from  the  task  force  method. 
Continuity    of   individuals   is      the    biggest    difference    between 
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the  two  methods.  Where  a  task  fores  reviews  one  project,  a 
committGe  will  be  convened  for  the  purpose  of  reviewing  many 
projects.  This  says  to  project,  managers  that  their  projects 
will  be  reviewed.  The  permanent  aspect  of  the  committee 
indicates  a  higher  degree  of  management  support  -han  a 
specially  convened  task  force.  As  with  a  task  force,  a 
permanent  committee  has  the  problem  of  the  am^ant  of  time 
reviewers  can  devote  to  projects  ander  review  while  still 
maintaining  their  workload.  Another  negative  aspect  is  that 
it  still  is  just  a  committee  and  will  lack  the  authorita- 
tiveness    of   a    function  staffed    with    full    time    personnel. 

The  fourth  method  is  a  combina'ion  of  full  -ime  and 
part  time  personnel.  This  method  can  be  accomplished  with 
the  use  of  one  or  more  full  time  personnel  to  maintain 
continuity  of  the  quality  assurance  function  and  augmented 
by  part  time  personnel  to  assisz  in  reviews.  The  obvious 
advantage  would  be  the  ability  to  add  specialized  knowledge 
as  needed  to  review  projects.  If  this  method  is  used,  one 
individual  should  be  named  to  head  the  quality  assurance 
function.  He  should  be  a  s-rong  personality  wi-h  supior 
knowledae  of  the  requireon  ents  of  of  a  quality  assurance 
function  and  possess  the  ability  to  direct  part,  time 
personnel  in  the  most  effecient  use  of  their  time.  It  is 
important  that  the  manager  be  on  equal  footing  with  other 
line  functions  and  can  insist  that  only  the  best  people  be 
assigned    to   quality    assurance. 

3 •      Reviews    and    Audits 

Reviews  are  conducted*  sequentially  throughout  the 
lifecycle  in  order  to  facillitate  transition  into  a  subse- 
quent developmental  phase.  As  previously  stated,  reviews 
should  occur  at  the  completion  of  each  development  phase. 
Perry  [ Ref .  4]  has  further  identified  twelve  review  points 
which  will  not  only  review  but  influence  systems  design  as 
well.      These   reviews    occur    at   the    following    points; 
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1.  Mid  justif  ication    phase,       ■ 

2.  End    of    justif icat iDQ    phase, 

3.  Business    system  solution    phase, 
u.      Computer    equipment    selection, 

5.  Computer    system  design, 

6.  Program   design, 

7.  Testing  and   conversion   planning, 

8.  Program   coding  and  testing, 

9.  Detailed    *est    plan, 

10.  Test   results, 

11.  Detail    conversion   planning   ani    programs,    and 

12.  Conversion  results. 

Naturally,  the  number  of  revisw  points  would  depend 
on  a  number  of  variables  including  size  of  system,  impact  on 
-he  organization,  and  makeup  of  the  quality  assurance 
organization.  In  addition  to  these  review  points,  quality 
assurance  can  perform  valuable  consultation  while  conducring 
the    reviews. 

Hos-^  authors  are  not  as  specific  as  Perry  as  to  the 
tiling      and      placement:      of         review      poinrs.  The      general 

consensus  is  that  reviews  must  be  predefined,  occur  at  key 
points  in  the  development  process,  be  understandable  and 
thorough,  and  are  conducted  in  accordance  with  prescribed 
procedures. 

There  are  a  number  of  techniques  that  a  quality 
assurance  review  team  may  employ  during  the  course  of  the 
review,  When  gathering  information  about  the  system  being 
reviewed,  such  things  as  project  docamentat ion,  sys-em  docu- 
men-^ation,  interviews,  observations,  and  the  use  of 
established  checklists  are  appropriate  methods  of  gathering 
information.  Practices  used  when  attempting  to  validate  the 
information  given  during  the  gathering  phase  inculde 
testing,  evaluating  test  iata,  foraulating  base  case  data, 
and      individual      confirmation.         Aftsr      the      information      is 
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gathered  and  validated  qaality  asoucance  must  evalua~e  -^he 
data  fcr  management.  This  evaluatiDn  is  typically  based  on 
incuitiive  and  evaluative  judgement,  mathematical  simalazion 
or  modeling,  expert  advica,  or  guantitatiive  analysis.  This 
list  is  not  exhaustive,  bat  is  rspresantative  of  The  types 
of  technigues  used  by  quality  assurance  reviewers  in 
achieving   fair  and   comprehensive    ra  views. 

Auditing  is  sometimes  dif f  er entia*' ed  from  reviews. 
Audits  are  usually  thought  to  be  final  acts  where  all  loose 
ends  in  a  quality  program  are  ti=d  up.  Types  of  audits 
include  in^house  audixs  where  the  audi^  verifies  that  the 
developer  is  adhering  to  all  development  standards  and 
procedures,  subcontractor  audits  to  ensure  that  the  subcon- 
tractor is  complying  with  all  software  standards  and 
procedures  imposed  by  the  contract,  and  fact-finding  audits 
in  which  the  subcontractor  is  evaluated  to  ensure  he  is 
capable  of  furnishing  reliable,  quality  software  of  the  type 
deemed   necessary  to    meet    contractural    requirements. 

The  Institute  of  Electronic  and  Electrical  Engineers 
standards  propose  a  certain  minimum  number  of  reviews  which 
should  be  conducted  during  the  software  developm<=nt  life- 
cycle  [Eef.  7].  These  include  a  software  requirements 
review  to  ensure  the  adequacy  of  the  requirements  stated  in 
the  software  requirements  specifications,  a  preliminary 
design  review  to  evaluate  the  technical  adequacy  of  the 
preliminary  design  of  the  software,  and  a  critical  design 
review  to  determine  the  acceptability  of  the  detailed  soft- 
ware designs.  Recommended  audits  consist  of  a  functional 
audit  which  is  held  prior  to  software  delivery  to  verify 
compliance  to  all  requirei ents  specifications,  a  physical 
audit  to  verify  that  the  software  and  documentation  are 
internally  consistent  and  ready  for  delivery,  and  in-process 
audits  to   verify   consistency  of   the    design. 
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Verification    is      another   word      for   testing.  It    is 

essentially  ensuring  that  the  conditions  are  as  stated.  It 
involves  doing  whatever  is  nscessary  to  verify  that  the 
statements  or  conditions  are  correct.  Although  correctness 
is  the  overall  goal  for  most  testing  efforts,  it  is  not 
always  the  overriding  concern.  Larg=  programs  are  sometimes 
so  complex  that  they  never  completsl/  satisfy  their  specifi- 
cations. These  programs  may  be  quite  usable  because 
failures  are  encountered  infrequently  in  practice,  and  when 
thay  ^o  occur,  th>=ir  impact  en  a  user  is  sinall.  To  b=r 
usable,  correctness  is  not  always  necessary  and  sometimes 
not  good  enough,  A  correct  program  may  sa-^-isfy  a  narrowly 
drawn  specification  and  yet  not  bs  suitable  for  operational 
use  because,  in  practice,  inputs  not  satisfying  the 
specification  are  presented  to  ths  program  and  r9sults  of 
such  incorrect  usage  are  unacceptable.  Thus,  if  a  proaram 
is  correct  with  regard  to  an  inadsquate  specification,  its 
correctness  is  of  litxle  value.  The  problem  which  arises  is 
that  most  testing  consists  of  correctness  tests.  There  is 
very  little  testing  done  for  reliabili-^y,  robustness,  effi- 
ciency, and  other  properties  which  nake  a  valuable  software 
product.  Whatever  property  is  being  tested,  the  tests  which 
are  valuable  are  those  where  the  result  is  not  predictable, 
so  that  application  of  the  test  and  acquisi-ion  of  the 
result  constitutes  an  information  gain  or  a  reduction  in 
uncertainty  [Ref.  26].  To  achieve  this  goal,  tests  should 
check  the  program  at  the  boundaries  of  its  behavior.  In 
order  for  software  to  be  tested  in  the  most  effecient 
manner,  a  test  plan  with  complete  procedures  and  methodolo- 
gies must  be  formulated.  Dther  than  from  the  obvious  reason 
of  ensuring  test  efficiency,  the  reasons  for  this  are  to 
provide   an      audit  trail   of      testing   so    -hat      future    problems 
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may  be  dissected  from  the  point  they  initially  surfaced  and 
that  boundary  areas  where  test.inq  would  be  aos^  -fficient 
may    be   easier    to    identify. 

Software  quality  assurance  should  become  involved  in 
testing  in  a  number  of  areas  as  illustrated  in  Figure  2.12 
Before  the  testing  begins,  it  should  ensure  that  all  soft- 
ware, hardware,  and  the  environment  are  in  a  satisfactory 
state  and  that  test  and  simulation  software  have  been 
defined  and  are  under  control.  It  should  witness  loading 
and  running  of  the  software  and  ensare  the  test  results  are 
retained  and  all  discrepancies  notsd.  Finally,  quality 
assurar-ct  should  ass  is~  -;.iLne  zunc~iDr.s  zd  ^nours  cis".  iririi.r.a~ 
tions    concerning   deviations    and    discrepancies   are    recorded, 

1111-5-5  2  779  A ,  the  standards  for  software  development 
used  by  the  Department  of  Defense,  contains  a  comprehensive 
list      of      software      testing       proce3ures      [Ref-    22].  These 

testing  procedures  are  utilized  by  lany  civilian  software 
development  oraanizations  and  consist  of  the  following:  (a) 
analysis  of  software  requirements  to  determine  testability, 
(b)  review  of  test  requirements  aai  criteria  for  adequacy, 
feasibility,  and  traceability  and  satisfaction  of  require- 
ments, (c)  review  of  test  plms,  procedures,  and 
specifications  for  compliance  with  contractor  and  contrac- 
tual requirements  and  to  insure  that  all  authorized  and  only 
authorized  changes  are  implemented,  (d)  verification  that 
tests  are  conducted  in  accordance  with  approved  test  plans 
and  procedures,  (e)  certification  that  test  results  are  the 
actual  findings  of  the  tests,  (f)  review  and  certification 
of  test  reports,  (g)  ensuring  that  test  related  media  and 
documentation  are  maintained  to  allow  repeatability  of 
tests. 
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Before  Test 

) 

1.  All  software  and  hardware  under  control 

2.  All  test  and  simulation  software  defined 

3.  All  facilitiss  available 

During  Test 

U.   Witness  loading  and  running 

5.  Reccrd  discrepancies 

6.  Identify  and  retain  output  and  results 

After  Test 

7.  Participate  in  Analysis 

8.  Raise  outstanding  points  as  discrepancies 

9.  Retain  analysis  rs  3ults 

10.   Certify  tes-  r  eoort  on  sa-^.isfactory  comolet: 

! 

-oni 

1 

1 
1 

Figure  2.12   Typical  Functions  of  a  QA  Dept.  During  Testing. 

5«   Software  Quality  Assurance  TddIs  and  T^chniaues 

Imoroving  software  development  and  test  processes 
depends  in  large  part  upon  the  application  of  proper  tools 
and  technigues  to  the  development  lifecycle.  The  differen- 
tiation between  tools  and  techniques  is  very  clear.  A 
technique  may  be  defined  as  a  procedure  for  implementing  a 
reliability  or  quality  goal.  Techniques  consist  of  stan- 
dards and  procedures  used  in  development  and  maintenance  of 
software  systems  [  Ref  -  25].  Such  things  as  structured 
programming,  top-down  design,  system  modularity,  proper 
language  selection,  abstraction,  information  hiding,  and 
program  design  languages  are  generally  thought  to  be  techni- 
ques in  achieving  software  quality. 

Tools,  en  the  other  hand,  have  been  defined  as  an 
automated  technique  [Ref.  25].  CorapuLer  programs  which 
perform  measurement  tasks  which  would  otherwise  have  -^o  be 
done   manually  are   considered  tools.     There   are  a   large 
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number  cf  tools  -.o  ensure  software  quality  assurance  on  th  = 
marketplace.  The  problei  is  that  most  automated  testing 
tools  are  expensive  to  install  and  use,  different  test- 
require  very  different  tools  and  most  tools  are  incompatible 
with  each  other  [ Ref •  27].  Selection  of  specific  tools 
should  be  done  only  after  careful  analysis  of  the  objectives 
desired  of  the  tools,  the  tools  cost-  funding  and  the  criti- 
cality  of      the  software      functions   to      be   tested.  Another 

consideration  should  be  the  phase  of  development  which  the 
software    is    in.         Figure    2.  13   shows    some   typical    tools    which 
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Figure    2.13        Tools    Used   by   Software   Phase. 

may  be  used  during  the  lifecycle  of  software  development. 
While  not  a  specific  list  of  tools  which  may  be  utilized, 
the    figure      gives  an      idea    of      the   types      of   tools      found   in 
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industry  today.  A  brief  explanation,  of  selected  tools  from 
the      above      list      follows.  System      modeling--a      techr.ique 

whereby  a  simulation  of  the  hardware/software  system  is 
programmed  using  a  simulator.  Interactions  between  hard- 
ware, software,  and  perso[inel  are  simulated  and  incompatible 
system  requirements  often  become  evident  after  system 
modeling.  PSL/PSA      (Problem        Statement      Language/Problem 

Statement  Analyzer) --^This  is  a  specific  tool  licensed  by  the 
University  of  Michigan,  Project  ISDOS.  It  provides  a  means 
for  describing  information,  computer  and  software  systems. 
A  requirements  data  base  built  from  several  contributers  can 
be  checked  for  consistency  and  formal  completeness.  Design 
modeling — Critical  algorithms  are  coded  in  a  representative 
manner  to  determine  of  the  design  will  result  in  the  desired 
accuracy      and   execution      times.  Timing,      memory      margins, 

resource  utilization,  and  traffic  rates  are  modeled  to 
ensure  adequacy.  Requirements  traceability  tool--Sof tware 
requirements  are  linked  to  successive  design  data  base 
entries,  test  planning,  and  tesr  ia-^a  entries  to  provide 
requiremen-s  traceability.  Interactive  debug  tools — The 
debug  tool  con-rols  the  code  while  ir  is  executed  and 
displays  memory  and  registers.  The  registers  and  memory  can 
be  displayed  while  the  code  is  executed  instruction  by 
instruction.  Preset  memory  locations  and  registers  hold  any 
desired  value  thus  allowing  branches  to  be  executed  and  the 
logic        debugged.  ICS         (interpretive      computer        simula- 

tion)— This  tool  allows  the  instructions,  interrupts  and 
input/output  capabilities  to  be  made  visible  by  simulating 
the      architecture  and     memory   of      a      larger    computer.  The 

program  can  be  started  and  stopped  in  order  to  evaluate 
performance     of      the      program   at      various      points.  Stress 

tests — As  the  name  implies,  this  tool  tests  the  computer 
under  worst  case  conditions  of  various  parameters  such  as 
memory  input  rates,  memory  utilization,  etc. 
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Hardware/software  test  beds— --a  test  bed  is  the  system  le  ^/el 
hardware  joined  with  the  system  software  and  combined  with 
xhe  appropriate  test  drivers,  monitors  and  env ironmemial 
simulators  to  provide  as  near  an  operational  system  as  is 
possible.  Regression   tasting — Regression      testing    ases      a 

standard  proven  test  for  testing  the  software  after  a  change 
has  been  incorporated  in  the  softwars  in  order  to  detect  any 
side  effects  or  errors  due  to  the  change.  These  tools  are 
but  a  few  of  the  literally  hundreds  on  the  market.  Tools 
can  be  a  valuable  and  useful  aiditioi  to  any  software  devel- 
opment life  cycle  but  must  be  chosen  carefully,  checked  out 
before  commitment,  and  used  in  the  proper  perspective.  This 
means  that  rhe  tool  should  be  recogiized  as  a  tool  and  rhe 
results  should  be  evaluated  carefully  and  ac-iDn  only  taken 
on   specific   results    generated   by    the    tool. 

6.      Software    Document at  ion 

Perhaps  the  weakest  link  in  nodern  software  develop- 
ment has  been  documentation.  There  are  a  number  of  sources 
of  information  which  provide  guidelines  concerning  sof-ware 
documentation.  MIL-S-52779^  and  th=  IEEE  quality  assurance 
standards  both  provide  requirements  for  documentation. 
MIL-S-52779 A  calls  for  all  documentation  standards  and 
programing  conventions  and  practicas  to  be  ased  for  all 
software  to  be  referenced  in  the  quality  assurance  plan. 
The  IEEE  standard  calls  for  identification  of  the  documenta- 
tion governing  the  development  and  verification  of  the 
software  and  an  explanation  of  how  the  documents  are  to  be 
checked.  It  further  calls  for  a  number  of  specific  docu- 
ments. These  include  a  software  requirements  specification 
(SRS)  ,  software  design  description  (SDD)  ,  and  software  veri- 
fication plan  (SVP)  .  PIPS  PUB  38  provides  extensive 
guidance  concerning  documentation  of  computer  programs  and 
ADP      systems.           Software      documentation         is      an      extremely 
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critical  aspect  of  software  developaent  in  rhat  it  is  the 
means  cf  communication  which  the  issigner  uses  with  his 
colleagues,  management  and  the  technical  authorities  of  the 
customer.  Although  it  is  widely  recognized  that  good  docu- 
mentation practices  should  be  maintained  for  all  system 
projects  and  there  are  ample  guidlines  with  which  to 
proceed,  documentation  remains  largely  inadeguate.  Much  of 
it  is  old,  it  is  poorly  written  or  written  in  such  a  manner 
as  to  be  incomprehensible  to  the  average  reader,  it  may  not 
be  thorough  or  leave  out  elements  which  are  critical  to  the 
software  in  guestion,  or  it  hasn't  been  changed  to  reflect 
current  practices.  Software  quality  assurance  must  realize 
the  reguirement  for  good  documentation  and  take  steps  to 
ensure  that  documentation  which  accompanies  developed  soft- 
ware   is   complete,   clear,    accurate   and    concise. 

7  •      Configur  at ion    Management 

Configuration  management,  consists  of  identifying  ih- 
configuration  of  the  software  systern  at  discrete  points  in 
time.  The  purpose  is  to  systematically  monitor  changes  to 
this  configuration  and  maintain  ths  integrity  and  trace- 
ability  of  this  configuration  throughout  the  system  life 
cycle.  Ix      is      primarily         concerr-ed      with      ensuring      the 

integrity  and  continuity  of  design.  Quality  assurance, 
through  configuration  management,  should  enforce  the 
following:  (a)      Configuration      Identification- A    sys-em      of 

recording  the  technical  description  of  individual  computer 
programs  and  supporting  doc umenration,  thus  documenting  the 
functional  and  physical  characteristics  of  the  configured 
item.  (b)  Configuration  Cd ntrol-applies  to  configured  soft- 
ware and  documentation  after  they  have  been  released.  It 
also  provides  a  control  for  changes  and  library  features. 
(c)  Configuration  Status  Accounting-t he  recording  of  the 
status   of      the   system's   configuration..  The   purpose      is    to 


51 


know    exactly   what  the  current     configuration   is,      and   hew 
was    achieved. 
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III.    A    SORVSY    OF    MI  LIT  MI     AND    CIVILIAN    SOFTWARE    ^UALIll 

1SS3E4ICE    PROGRAMS 

A.       INTRODOCTIOH 

As  stated  in  [  Ref.  22]  the  purpose  of  a  quality  assu- 
rance program,  "Is  to  assure  that  software  developed, 
acquired  or  otherwise  provided  under  the  contract  complies 
with  the  requirements  of  the  contract".  Another  definition 
of  quality  assurance  is,  "A  planned  and  sys^eniatic  pat-em 
of  all  actions  necessary  to  provide  adequate  confidence  that 
the  items  will  perform  satisfactorily  in  actual  operation" 
[Raf.  28].  Still  another  definition  is,  "The  early  detec- 
tion and  correction  of  deficiencies  and  the  evaluation  of 
overall   quality    performance"   [Ref.    23].  Although    pros    and 

cons  on  the  merits  of  each  of  these  definitions  can  be 
thought  of  easily,  they  all  have  the  same  goal  -  prevent 
customer   complaints. 

Quality  assurance  achieves  this  goal  through  control. 
This  control  function  is  based  on  rhe  existence  of  some  type 
of  plan  and  the  control  function  is  simply  ensuring  adher- 
ence to  that  plan.  Effective  control  will  detect  deviation 
from  the  plan  early,  before  it  actually  occurs.  Ineffective 
control  detects  deviation  a s  it  happens,  when  it's  too  late. 
Two  key  factors  in  effective  control  are  (1)  total  knowledge 
of  the  plan  and  (2)  establishment  of  milestones  against 
which  progress  on  the  plan  can  be  measured.  3y  monitoring 
these  milestones,  action  can  be  initiated  to  prevent  poten- 
tial   deviation   from    the   plan. 

In  essence,  quality  assurance  is  not  something  that  can 
be  added  later  in  the  software  development  process.  It  is 
not  the  job  of  a  single  person  or  group  of  persons  to  see 
that    quality      is   added  at      the    right      time    and   in      the   right 
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amount.  Qualify  assurance  begins  at  The  s-^art  of  th^ 
developmenx  process  and  is  continualiy  added  at.  each  s-ep 
along  ^he  way.  The  primary  job  of  the  quality  assurance 
group  then  becomes  the  development  of  the  quality  assurance 
plan  and  once  it  is  developed,  the  managemen*:  of  it 
throughout    the   development    process. 

The  next  sections  of  this  chapter  will  discuss  standards 
which  are  in  use  for  developing  quality  assurance  plans. 
Following  that  will  be  a  discussion  of  the  typical  software 
development  cycle.  The  final  section  of  the  chapter  will 
discuss  the  quality  assurance  programs  in  use  at  two  major 
civilian    companies    and  at    the   iJaval    Dcean    Sys-ems    C-in-:er. 

B.       SOFTWARE    QUALITY    ASSURANCE    PLAN    STANDARDS 
1.      Milirary   Specification   5 2229 A 

Mil-S-52779A  applias  to  the  acquisition  of  software 
either  alone  or  as  par-  of  a  complete  system.  It  requires 
the  establishment  and  implementation  of  a  software  quality 
assurance    program  by   the    contra  otor. 

Paragraph  3.2  of  Mil-S-52779A  deals  with  the  soft- 
ware guality  assurance  plan.  According  to  this  specification 
[Bsf.  22]  any  software  quality  assurance  plan  will  include 
the    following  areas: 

1.  Tools-  Techniaues  and  M ethodDlcgies:  What  are  they 
and  how  will  thay  support  the  overall  Quality 
Assurance  Program?  Examples  include:  Operations 
Research  -  Systems  Analysis,  functional  and  perfor- 
mance requirements  analysis,  error  analysis,  software 
optimization  tools,  specification  tracing  and  coding 
conventions. 

2.  Computer  Program  Design:  Hdw  will  design  logic, 
fulrillment  of  reaiii  rements,  completeness  and  compli- 
ance  with  specified    standards   be    evaluated? 

3.  Work  Certification:  How  will  the  description,  author- 
ization and  completion  of  work  be  certified  or 
approved? 

4.  Documentation:  What  documentation  standards  and 
program  conventions    will    be  used? 
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5.  Computer  Erogram  Library  Con-^-rois:  How  will  diifersnt 
coKiDU-er  prcgram  v^rsior-S  br  identified  and  docu- 
mented? The  objective  her?  is  to  insure  that  only 
approved  niodif icario  ns  ire   made    and    implemented. 

6.  Reviews  and  Audits:  How  will  rev j.ews  and  audits  be 
conducted  to  insure  traceability  from  initial 
requirements   to  final   product? 

7.  Configuration  Management  (CM) :  How  are  Software 
Quality   Assurance   and  CM    related? 

8.  Testing:    This    section  includes    the    following   areas   - 

a)  Analysis   of  ragairements   to    determine    testability 

b)  Review  of  test  requirements  and  criteria  t.o  insure 
adequacy,  feasability,  traceability  and  satisfac- 
tion  of  reguir emen"^ s. 

c)  Review  of  test  plans,  procedures  and  specifica- 
tions for  compliance  wi-.i  ::cr.  tr  actual  re4uirefr;-r.ts 
and  to  insure  all  authorized  changes  are  imple- 
mented. 

d)  Verificaticn   of   tests. 

e)  Certification   of    test    results. 

f)  Review   and   certification   of    test    reports. 

g)  Maj-tenance  of  test  material  to  insure  repeat- 
ability. 

h)  SuDport  software  and  hardware  used  during  develop- 
ment   must    be  acceptable    to    the    government. 

2-      USE   Standard   for   Software   ^ualit^   Assurance    Plans 

The  purpose  of  this  standard  is  to  provide  uniform, 
minimum  acceptable  requirements  for  preparation  and  content 
of  Software  Quality  Assurance  Plans.  The  standard  applies 
to  the  development  and  maintenance  of  critical  software 
(i.e.  where  failure  could  impact  safety  or  cause  large 
financial  or  social  losses).  For  non-critical  software  or 
software  already  developed  a  subset  of  the  requirements  may 
be    used   [Ref.    23  ]. 

The  following  are  the  major  sections  and  subsections 
of  the   plan   as   outlined   in    [Ref.    23], 

1 .  Purpose. 

2.  Reference   Documents. 

3.  Management. 

a)    Organization 
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b)  Tasks 

c)  Responsibilities 
U.      Dccumentation 

a)  Purpose 

b)  MinimuiD  Required  Documentation:  Software 
Requirements  Specification,  Software  Design 
Description,    Software    Verification    Plan. 

c)  Othei;:  Computer  Program  Development  Plan, 
Configuration  Management  Plan,  Standards  ana 
Procedures    Manual, 

5.  Standards,   Practices    and    Conventions. 

a)  Purpose 

b)  Content:  Document  Standards,  Logic  Structure 
S^^andards,    Coding    Standards,    Commen'-ary    S^andrds, 

6.  Reviews   and   Audits. 

e)    Purpose 

b)  Minimum  Rquirem=nts:  Software  Requirements  Review, 
Preliminary  Design  Review,  Critical  Design  Review, 
Functional  Audit,  Physical  Audit,  In-Prccess 
Audits  . 

7.  Configuration    Management. 

8.  Problem   Reporting   and  Corrective    Action. 

9.  Tools,    Techniques    and   Methodologies. 

10.  Cede   Control. 

11.  Media   Control. 

12.  Supplier    Control. 

If  any  of  the  above  sections  are  not  pertinent  to 
the  project  for  which  the  plan  is  being  written,  a  statement 
stating  this  non -applicabil ity  should  be  included  under  the 
section  heading  along  with  reasons  why  it  is  not  applicable. 
If  additional  sections  are  needed  they  may  be  included  at 
the    end. 

C.       PHASES    OF    THE   SOFTWARE    DEVELOPMENT    PROCESS 

It  is  generally  agreed  upon  that  the  software  develop- 
ment process  consists  of  at  least  th=  following  seven  phases 
[Ref.    24]    :      Conceptual,         Requirements    Definition,      Design, 
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Coding  and  Checkout,  Test-ing,  Intagration,  and  Operaticnal. 
Software  definition  takes  place  in  the  Conceptual  phase. 
This  consists  of  feasability  stadias,  t.rade-off  s-udies  and 
analyses  to  define  specific  requiramants  to  be  allocated  to 
computer  resources.  Once  these  requirements  are  defined  and 
documented  they  form  the  basis  for  a  draft  systam  specifica- 
tion   which    will    be    used   during  the   following   phases. 

During  the  s«=cond  phasa,  Requiraaents  Definition,  it  is 
determined  which  system  raquiraments  will  be  implemented  by 
software.  Through  analysis  it  is  datermined  which  software 
functions  are  needed  and  the  inputs,  processing,  and  ou-puts 
that  ars  required  for  each  function.  Also  part  of  this  phase 
is  the  finalization  of  the  systam  specification  and  the 
praparation  of  the         draf-  software  requiremen-s 

specification. 

Following  the  Requiremants  Definition  phase  is  the  third 
phase  of  the  development  cycle.  Design.  The  object  of  -his 
phase  is  to  come  up  with  a  software  design  that  will  impla- 
ment  the  functions  identified  luring  the  Requirements 
Definition  phase.  The  design  will  include  actual  algorithms 
and  equations  along  with  control  logic  an  data  operations  to 
be  performed.  The  finalization  of  the  software  requirements 
specification  and  the  praparation  of  the  draft  software 
design   specification    will   also    take    place    during    this    phase. 

The  fourth  phase.  Coding  and  Checkout,  includes  trans- 
lating the  software  design  into  a  computer  programming 
language.  Usually  it  is  a  high-order  language  but  it  may 
also  be  assembly  language.  Once  compilation  and  assembly 
errors  are  corrected  each  individual  program  module  is 
executed  to  remove  obvious  errors.  This  procedure  consti- 
tutes  the   checkout. 

Once  coding  is  complete  the  fifth  phass  of  the  cycle. 
Testing,  begins.  Here  the  software  which  has  been  developed 
is  tested  to  shew  that  it  is  consistent  with  system  and 
software   requirements. 

57 


During  Integration,  hardware  and  software 
together  and  system  and  opera 'ioaal  ^as-cing  is  conducted. 
The  object  of  the  testing  is  to  ir.sare  the  satisfaction  of 
system   requirements    in  the    actual   or    simulated   environment. 

The  last  phase  of  the  cycle  is  the  Operational  phase. 
The  software  has  been  accepted  for  use  and  the  only 
remaining  activities  are  maintenance  and  modification. 
Figure   3.1    [Ref,    24]   shows      the   software    development    process 
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Figure    3,1        Software   Development   Process. 

and  the  key  outputs  of  the  phases.  Figure  3.2  [Ref.  24] 
shows  how  the  quality  assurance  activities  fit  into  the 
development  process.  Figure  3.3  [Ref.  29],  although  it  does 
not  specify  all  seven  phases  decribed  above,  it  dees  show 
the    activities    of      not   only    the    Quality      Assurance    group    but 
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Figure  3.2   Softwars  Quality  Assurance  Process. 


of  the  groups   in  the  proiact  organization.    The 
listed  are  all  quality  assurance  related. 


.vizies 


D.       QUALITY    ASSOBANCE    PROSHAMS 

1 .      TRW   Defense    Systems    Group 
a.      Background 

Kurt    F.     Fischer    [Ref.    30]    writes; 


At    TRW      Defense  and   Space    Systems      arouD    the  need      for    a 
division   wide    crganizaiion   zo    assist    software   management 

"    -.-_-•---    .,  -      La-^e    1960s.       Like    most 

'as  concerned  about  the 
frequent  cost  overruns'  and  schedule  slippages  in  its 
software  projects,  and  decided  to  develop  methods  to 
turn  this  trend  around.  As  part  of  that  decision,  it 
established   its  first    qua lity ' assurance    staff    in    19b9. 


became   quite   apparent   during    the 
other      software  vendors,         TRW    wi 
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At  TEW  the  software  quality  assurance  functions 
ars  performed  by  an  crgaaiza-icn  ti~led  Product  Assurance. 
This  organization  is  headed  by  a  vic=-  president  level  sraff 
director.      Figure      3.4  [Ref.    30]      illustrates   the      corporate 
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Figure   3.4        TRW   Corporate  Organizational   Structure. 

organizational  structure.  The  Product  Assurance  organiza- 
tion actually  perforins  a  dual  role:  Quality  Assurance  and 
Configuration  Management.  The  reason  for  this  is  that  both 
areas    have   been    found   to    share    similir    characteristics. 

1.  They    perform    s-aff   Driented   functions. 

2.  The  perfgrmance  of  their  functions  is  often  times 
more  credible  when  done  by  an  independent  organiza- 
tion. 

3.  Staff  personnel  share  many  aptitudinal  characteris- 
tics (e.g.  close  attention  to  detail,  preference  for 
wide    visibilty  tasks)    [Ref.    30]. 
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Product  Assurance  on  ~he  product  level  is  headed 
by  an  Assistant  Program  Minager  for  Product  Assarar.cs  who 
has  responsibility  for  both  Quality  Assurance  and 
Configuration  Management.  The  APM  for  Product:  Assurance 
receives  his  direction  from  the  project  manager,  yet  he  and 
his  staff  remain  organizationally  independent  from  the 
project  by  reporting  functionally  to  the  Product  Assurance 
organization  at  the  corporate  level.  Figure  3.5  [Ref,  30] 
illustrates  the    project   organizational    structure. 

b.      Quality    Assurance  Objectives 

objectives,    the    following   activities   [Ref.    31]   are    performed 

by    the   Quality    Assurance    group   at   TR^: 
1. 


ment  specifications  and  the  t=st  olan  and  from  there 
TO  test  procedures.  QA  will  additionally  insure  that 
all  requirements  are  traceabl?  *o  the  product  speci- 
fication, 

2.  QA  will  develop  and  maintain  a  sof-^ware  reauirements 
matrix.  This  matrix  will  be  maintained  throughout  the 
software    development    cycle. 

3.  All  documentation  generated  iuring  the  project  will 
be   reviewed    by   QA    personnel. 

U.      Conduct   audits   of    the    software    development    process. 

5.  QA  personnel  will  participate  in  all  formal  reviews 
and  audits  (.e.g.  Software  Requirements  Review, 
Preliminary  Design  Review  and  Critical  Design 
Review) , 

6.  Insure  the  implementation  of  built  in  checks  and 
balances . 

7.  QA  oersonnel  will  monitor  and  witness  the  Preliminary 
Qualification  Test  (PQD  and  Formal  Qualification 
Test  (FQT),  These  test  results  will  be  cosianed  by  QA 
personnel.    All    QA   test    records    will    be    maintained. 

8.  QA  personnel  will  monitor  the  Configuration 
Management  practices  during  software  development. 
They  will  also  test  to  insure  the  integrity  of  the 
software    configuration. 

9.  QA  personnel  will  participate  in  all  Configuration 
Change  Board  (CCB)  meetings  and  orovide  a  review  of 
all  proposed  changes  in  the  software  development  and 
test  Drocess  to  again  insure  the  integrity  of  the 
software    configuration. 
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10.  QA    will      support    the    developmant    of      project   Software 
Standards   and    Procedures. 

11.  Verify   that    all  requirements    5Lnd    functional    capabili- 
ties  nave  been   satisfied    by   software   testing. 

12-    QA    personnel    will   insure      that    the    delivered   software 
package  meets    contractual  requirements. 

13.    A    system    for    tracking      Software    Problem   Raoorts     (SPR) 
and    Design  Problem   Reports    (DPR)     throughout    the    soft- 
ware     development         and      production        life      will        be 
developed   and    maintained    by    QA    personnel. 
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1'4. 


To  insure  the  necessary  implsmentation  and  support 
fcr  all  QA  and  CY!.  iz  ti'T:.zt  =  s ,  Quali-v  Assuranc-^^  will 
participate    in   the   selection      orocess^of   all   software 


particip 
tools. 


c.      Quality    Assurance  Planning 

The  successful  implementation  of  an  effective 
Quality  Assurance  Program  relies  heavily  on  quality  assu- 
rance planning  during  the  early  phases  of  software 
development.  At  TRW  QA  planning  is  accomplished  by  a 
complete  review  of  the  early  project  documentation.  Examples 
of  such  documentation  include  the  Contract  Statement  of 
Work,  System  Specifications  and  "Prt^i'ect  '^lan  !?. "^cng  o"^. hers. 
Once  this  initial  review  is  completed  the  Quality  Assurance 
Plan  is  prepared.  This  plan  contains  the  functions,  tasks, 
and  responsibilities  of  the  Quality  Assurance  group  and  also 
identifies  the  quality  assurance  tools  needed  to  insure 
software  quality  in  the  areas  of  accountability,  test- 
ability, usability,  ma int ainabili-y  and  reliability.  Upon 
completion  the  plan  is  reviewed  by  other  project  organiza- 
tions and  approved  by  both  the  Program  ?1anager  and  the 
customer.  Upon  approval  -ask  assignments  are  iiade  to  carry 
out  the  activities  outlined  in  the  previous  subsection.  As 
noted  in  [Ref.  30]  these  assignments  are  based  on  level  of 
effort   and   must    remain  flexible    to   adapt   to: 

1.  The   needs      of    the      current   phase      in   the      development 
life-cycle. 

2.  Shifts    in      attention    needing      areas    (-eg.         technical 
problems) . 

3.  Unexpected  demands   placed    on    Quality    Assurance    by   the 
Project   Manager. 

Once      the      QA      Plan        is      approved      the      Quality 

Assurance   polocies   and   procedures   are    written   describing   the 

methods   and   procedures      to    be  used    in      the    implementation    of 

quality    assurance  requirements  defined    in    the: 

1.  System   Specification. 

2.  Contract    Statement   of   Work. 
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3.  Project   Plan. 

4.  Qk   plan. 

d.      Software    Standards 

As  Kurt  F.  Fischer  points  out  in  TRsf-  30], 
"The  purpose  of  software  standards  is  to  improve  the  main- 
tainability and  readability  of  the  software  product".  TRW 
has  developed  a  comprehensive  and  detailed  program  to  deal 
with  software  standards.  According  to  [Ref.  30]  this  program 
has   been   successful    for  two    reasons: 

1.  Software  standards  are  not  dictated  from  the  execu- 
tive     offices    or        from  Quality      Assurance,       bur      are 

ievalcped      cut      of      clcs^        ::::  :!i:?.  anica-^. ir?.       iscT^g      th=' 
Design,    Development,     Tesr,    QA    and   Project    offices. 

2.  A  tool  has  been  provided  to  automatically  check  the 
software  aaainst  niost  cf  the  standards.  This  allows 
programmers  to  audit  themselves  so  that  there  are  no 
surprises    at    turnover   time. 

The  Software  Standards  and  Procedures  (SSP) 
[Rsf.  31]  document  contains  the  quali'^y  provisions,  instruc- 
tions and  standards  for  =ach  project.  The  SSP  deals  with 
standards  concerning  software,  firmware,  design,  development 
and    testing.    The   categori~s    of    standards    included    are: 

1.  Source   code    formatting   standards 

2.  Techniques  to  be  used  in  software/firmware  design, 
code,    test  and  update. 

3.  Standards  dealing  with  QA  toDl  development  and  their 
use   during  design   development    and   checkout. 

Waivers    and      deviations    serve   to      complement   the 

standards.      They     allow   permanent      oc    temporary      relief   from 

compliance   with    the    standards    due  to    technical   difficulties, 

inefficiencies   or  schedule    impact.    All    waivers    or    deviations 

must    be      approved  by      both    the      QA   tlanager      and   the      Project 

Manager. 
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e.      Quality    Assurance   Tools 

The  use  of  software  tools  should  only  be  consid- 
ered where  it  will  prove  to  bs  more  cost  effective  and  more 
accurate  to  have  the  task  auromated  rather  than  performing 
it  manually.  Tasks  which  may  fall  into  this  category  are 
often  tedious,  menial,  boring,  difficult,  error  prone, 
repetitive   and  costly  [Ref-    30]. 

Although  TRW  is  currently  using  improved  and 
updated  tools  the  following  ars  examples,  taken  from 
[Rsf.    30],    of   the  types   of    tools    in    use    at    TRW: 

1.  Product    Assurance   Confidence   Evaluator    (PACE)       -   PACE 

15    dssign-id    zz    quiiit  ita-i  V9ly    =valiat9    h;w    tncroaahlv 
and   rigorously  a    prDgram    has    been    tested. 

2.  FORTRAN  Code  Audiror  (FCA)  -  This  tool  audits  coding 
standards  and  by  doing  so  allows  enforcement  of  these 
same   standards. 

3.  Structured  Programmina  Auditor  (STRUCT)  -  STRUCT  is 
used  to  ensure  programs  comply  with  the  structured 
programming  standard.  It  is  executed  afier  PACE 
because  i"*:  relies  on  output  from  PACE.  It  evaluates 
program  structure  based  on  the  following  six 
constructs;  seqasnce,  if -then-else,  dc-while, 
do-unnii,    case,    esca pe-?rom-loop. 

U.       Units  Consistency   Analvsis      (OCA)       -   This    is      used   on 
FORTRAN   source   cod=    and   the    associa-^ed   data    base.      It 
:he      code   and    interprets    equations.         By    refer- 
in    zh- 
gnment 


f.      Quality    Assurance   Reviews    and    Audits 

Review  and  audit  points  are  established  during 
the  QA  planning  phase  to  ensure  that  design,  code,  inspec- 
tion,   testing,    and    documentation   are    compatible. 

As  used  at  TRW  reviews  s=rve  as  quality  assurance  critiques 
of  documents  while  audits  are  critiques  of  processes. 
Evaluation  criteria  for  documents  [Ref-  30]  consists  of  the 
following: 

1.  Adherence   to    format    and  pagination. 

2.  Clarity   of  objectives. 

3.  Technical   content. 
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u.   interdocument  consistency. 

5.   Traceability  to  higher  level  specifications. 

The  audits  conducted  by  Quality  Assurance 
personnel  [ Ref •  30]  serve  the  following  four  functions: 

1.  They  assess  compliance  of  source  code  and  documenta- 
tion TO  software  standards  and  procedures. 

2.  They  assure  traceability  of  requirements. 

3.  They  determine  tha  satisfaction  of  system  require- 
ments during  system  test  and  acceptance  phase. 

u.   They  assess  test  sufficiency. 

The  following  list  contains  the  audits  conducted 
by  rhe  quality  assurance  personnel  at  TRW.  A  brief  descrip- 
tion of  each  audit  is  also  included. 

1.  Unit  Development  Folder  (UDF)  Audits  -  The  UDF  is  a 
ncn- deliverab  le  item  which  provides  a  itechanism  for 
internal  control  and  also  provides  managemen-^  visi- 
bility for  software  de  velopoisnt.  The  UDFs  are 
prepared  and  maintained  for  each  software  unit 
provided  by  the  project.  The  definition  of  a  unit  is 
a  single  routine  or  a  group  of  logically  related 
routines.  The  UDF  is  a  collection  of  all  require- 
ments, design  data,  code,  and  test  data  pertaining  to 
a  sioecif  ic  unit.  The  UDF  serves  as  the  primarv 
surveillance  mechanism  for  the  auality  assurance 
oerscnnel  durina  the  design,  coding  ana  unit  test 
phases  of  the  software  development  project.  Each  UDF 
is  audited.  This  2. adit  is  divided  into  three  ohases 
in  an  effort  to  provide  early  detection  and  correc- 
tion of  possible  errors.  Eaoh  phase  is  designed  to 
audit  a  specific  a  r^^a  of  the  development  process. 
Following  is  a  description  [Ref.  31]  of  each  phase: 

a)  Phase  I  -  verifi=>s  appropriate  UDFs  have  been 
initiated,  proper  cover  sheets  and  inserts  have 
been  included,  '  requirements  have  been  stated  in 
the  requirements  section  and  that  the  design 
section  contains  the  current  working  design. 

b)  Phase  II  -  A  desk  check  and  automatic  code  audit 
is  performed  to  determine  if  code  isbeing  produced 
in  accordance  with  established  project  standards 
and  guidelines. 

c)  Phase  III  -  Verifies  that  each  UDF  audited 
contains  a  compilation  of  test  results  and 
analyses  necessary  to  demonstrate  that  the  unit  of 
code  has  been  debuaged,  init  development  testing 
is  complete  and  that  an  UD-to-data  design  descrip- 
tion exists. 

2.  Test  Data  Folder  ( TDF)  -  The  TDF  is  the  primary 
working  document  fjc  the  test  aroup.  As  such  it  also 
provides  a  surveillance  vehicle  for  quality  assurance 
personnel.  During  -^he  integration  and  acceptance 
testing  phases  of  the  development  project.  The  TDF 
contains  the  test  requirements,  test  plan,  test 
procedures,    test   execution   reports   and   software 
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prcblem  reports.  Quality  Assurance  reviews _the  feller 
iC  irsurs  Edscruacy  ccni't'leL^n^ss  snd  conf ortn'^. r.C''^  to 
standards  ""ouflineS  in"  €he  'Software  Standards  and 
Procedures  (SSP)  . 

3.  Ccnf igui;ation  Management  Audits  -  Quality  Assurance 
uses  xhis  audit  to  monitor  the  configuration  manage- 
ment activities  to  insure  rhat  they  comply  with  both 
the  CM  plan  and  documented  procedures. 

U.  Interface  Verification  Audi-  -  This  audit  is 
conducted  as  early  as  possible  in  an  effort  to  iden- 
tify and  correct  possible  interface  problems.  QA 
personnel  examine  the  requirements  design  and  program 
specifications, 

5.  Preliminary  and  Detailed  Design  Audits  -  Conducted 
prior  to  the  Preliminary  Design  Review  (PDR)  and  the 
Critical  Design  Review  (CDR)  respectively,  these 
audits  are  concerned  with  th=  format  and  content  of 
design  documentation  and  project  test  plans.  The 
results  of  these  audits  are  then  discussed  at  the  PDR 
and  CDR  respectively, 

6.  Independent  Quality  Audit  -  Not  only  do  the  Product 
Assurance  personnel  audit  the  various  areas  of  the 
software  development  project,  but  they  in  turn  are 
audited  by  the  corporate  Product  Assurance  organiza- 
tion, this  audit  do=s  in  fact  cover  the  whole 
project,  however,  the  QA  and  CM  areas  are  examined  in 
detail.  Upon  ccmplation  of  thr  audit  both  the  Project 
Manager  and  Assistant  Projeot  Manager  for  Proauct 
Assurance  receive  copies  or  the  audi,  report.  They 
also  receive  any  Corrective  Action  Reauests  (CAR) 
which  document  any  d  iscrepan.oi  9S  found  by  the  audit. 

7.  Audit  Reports  -  The  findings/results  of  the  above 
audits  are  provided  to  both  the  Project  Manaaer  and 
the  Assistant  Project  Manager  for  the  appropriate 
development  area.  In  additio:i  to  the  findinas  recom- 
mendations are  also  included,  A  periodic'  summary 
which  details  the  number/typ9  of  audits  performed, 
the  discrepancies  found,  all  corrective  action  in 
progress  cr  iorplemented  and  a  follow-up  on  corrective 
action  on-aoing  from  orsvious  reports  is  also 
prepared  by  Quality  Assurance  and  provided  to  the 
Project  Manager. 

As   previously  noted   at  the   beginning  of   this 

subsection  reviews  are  conducted   to  critique  documentation. 

»Jhat  follows  is   a  list  of  the  reviaws  conducted   at  TRW.   A 

brief  description  of  each  review  is  also  provided. 

1.  Software  Requirements  Revisw  -  This  review  is 
conducted  upon  completion  of  the  Software 
Requirements  Definition  phase.  QA  personnel  sit  as 
members  of  the  raview  board.  The  purpose  of  the 
review  is  to  insure  that  the  software  reauirements 
specifications  for  the  propossd  software  project  do 
in  fact  match  the  jsers  operational  requirements  for 
the  system. 


2.  Preliminai^Y  Design  Review  -  In  addition  to  conducting 
the  Preliminary  Design  Audit,  Quality  Assurance 
personnel   act   as  recordina   secretary   during   this 
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review.   The   PDR  [Ref.  31]  rsviews 

specification  fez    -lie  follr-^inq: 


each  develotsent 


3. 


a)  Traceability  of  reaui remsnt s  specification  to  the 
development   specification. 

b)  Requiremntes  satisfaction,  interface  definition 
and  specification    content. 

Critical  Design  Review  -  Ic  addition  "■  o  oerforming 
-he  Detailed  Design  Audit  Quality  Assurance  personnel 
serve  as  recording  secretary.  The  emphasis  or  the  CDR 
is  on  the  preparation  of  required  mat.erials,  briefina 
content  and  the  allocation  of  each  requirement  to  a 
functional  design   element   [Ref.    31]. 

4,  Design  Walkthroughs-  These  walkthroughs  are  held 
early  and  oftenduring  the  D=sign  phase.  The  walk- 
throughs are  conducted  by  technical  personnel  who  are 
taken  through  the  design  oa  a  step-by-st.ep  basis 
informally  by  the  designer.  This  done  in  ah  effort 
to  monitor  -he  consistsr.cy  or  th-B  dasicir.  apprcich, 
satisfaction  of  requireoien.t  s  and  completeness 
[Ref.    31  ]. 

g.      Test  ing 

In   [ Bef .    31]      it   states      that    Quality      Assurance 

has      review   authority      for      all      test    plans      and      procedures 

initiated   by   TRW   for   either      formal    or    informal    testing.      QA 

performs    a   selective   review    of      documentation   to    insure    that 

test    cases   and      test    procedures    direc'!  ly    correlate      with   the 

software    requirements.      The    two      prinary    functions    performed 

by      Quality      Assurance      dur ig   the      Testing      phase      are      test 

auditing      and      the      inspection   and      surveillance      of      formal 

tests.    Both   these  areas    are    described    below. 

1.  Test  Audit  -  This  audit  is  conducted  at  the  end  of 
each   test    phase  and    its    primary    purpose    is    to: 

a)  Assure  that  software  configuration  manaqement 
procedures    are   being   followed. 

b)  assure  that  the  test  specifications  beina  used  by 
the   test    group    are  the    current    approved    versions, 

c)  Assure  that  test  reports  identify  proper  test 
procedures  and  software  configuration;  specify  the 
test  analysis,  and  if  any  deficiencies  were  noted 
how   they    were    explained   and    accounted    for, 

d)  Verify  that  test  procedures  provide  a  step-by-step 
rationale  for  conductinq  a  test  and  that  test 
results  comply  with  acceptable  criteria  specified 
in    the   test    procedure. 

e)  Verify  that  Test  Data  Folders  comply  with  approved 
formats. 
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f)  Verify  .  compliaace  by  the  test  team  with  stated 
aianacj'?  tren-c  Drocs!  ures  for  ^hanae  control,  iiscrs- 
pancy    reporting,     and  rest    reporting  [Ref.    30]. 

2.  Inspection  and  Surveillance  of  Formal  Tests  -  This  is 
an  on-site  activity  performed  by  QA  personnel  during 
program   testing.    The    purpose    is    to: 

a)  Monitor  all  tests  to  ensure  that  the  actual  tests 
perforired  are  those  specified  in  the  documented 
test   procedures. 

b)  Assure  all  potential  discrepancies  are  recorded  in 
the   approved  manner. 

c)  Compare  configuration  of  hardware/software 
compen-^s  used  in  the  test  against  the  configura- 
tion   identified   in   the    etst    procedures. 

d)  Certify  that  analysis  of  test  results  is  correct, 
the  test  satisfies  the  intended  reguirements  ana 
accent  able  criteria  and  the  Test  Data  Folder  is 
complete. 

e)  Assure  that  master  copies  Df  test  procedures,  test 
results  and  test  reports  are  maintained  and  avai- 
lable   from    a   centralized   records    center    [Ref-    30]. 

3.  Test  Changes  -  The  test  director  may  make  corrections 
of  -typographical  errors  in  the  test  parameters  as 
long  as  they  do  n3t  deviate  from  specified  require- 
men-ts,  he  may  also  make  changes  in  the  test  set-UD 
and  in  addition  he  may  make  changes  in  the  procedure 
step  sequence  provided  that  tast  parameters  or  toler- 
ance accuracies  are  not  changed.  All  changes  must  be 
documented  and  will  be  reviewed  for  approval  by  th=- 
Test  Review  Board  and  Configuration  control  Board. 
Any  changes  causina  a  deviation  from  specified 
requirements  must  be  submitted  through  both  the 
Configuration  Control  Board  and  tne  customer 
CRef.    31  ]. 

4.  Test  Change  Request  (TC3)  -  This  is  the  vehicle  used 
for  requesting  changes  to  test  procedures.  A  descrip- 
tion of  the  problem  and  the  proposed  changes  are 
included  en  the  TCR .  Upon  approval  by  the  CCB  it 
provides  both  the  solution  to  the  problem  and  the 
means  for  implementing  that  solution.  Figure  3.6 
[Bef.    32]   is    a   sample   TCR    used    at    TRW. 

h.      Problem    Reporting  and   Review 

In    addition  to    the      problem    reporting    and    review 

procedures,      the    procedures      used   at    TRW    for      the    control    of 

changes   to    the   s eft ware    product    will    also    be   discussed.      The 

change   control   procedures    fit    in      at    this    point    because    more 

often      then   not      changes    are      made   in      response   to      problems 

which    arise   during   the  development   process. 

1.  Configuration  Control  Board  (CCB)  -  At  TRW  the 
Configuration  Control  Board  has  been  established  to 
review      and      approve      changes      to      documentation      and 
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Figure  3.6    Test  Change  Request. 
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softwar-.  Reguldrly  scheduled  meetings  are  held  but 
if  circu uistancss  warran-::  it  the  Proiect  'lanaaer  car. 
call  a  special  meeting.  Ths  Drojeci:  manager  aG~s  as 
chairman  and  oiembsrs  include  personnel  from  Product 
Assurance,  Data  Processing  '  Sys*"e!ns  Engir.'^'^ring, 
Apolications  Software,  Systems  and  Support ^Software, 
Data  Processing  Hardware,  and  Integra-ion  and 
Testing.  The  user  may  also  take  part  in  the  meetings 
and  in  any  event  will  be  supplied  a  copy  of  the 
minutes. 

The    following   are     the   documents    used    at      TRW    in 

an    effort   to   maintain  configuration      control   over    a   software 

development   project.        Th^se   documents    are   submitted      to   the 

CCB    for   its  review    and  approval. 

1.  Desian  Problem  Report  (DPR)  -  This  report  is  used  to 
reauest  changes  to  baselined  (already  approved)  docu- 
ments and  also  to  initiate  chanaes"  to  formally 
baselined  customer  controLlii  docuieLts.  rhe  DrJl 
contains  both  a  description  of  the  problem  and  a 
proposed  solution.  0nc9  it  has  been  approved  by  the 
Configuration  Control  Board  it  provides  -^he  accepted 
solu-^ion  to  be  implemented.  Marked  up  change  pages 
must  be  attached  to  the  DPR  "-.o  illustrate  the  changes 
being  made.  QA  personnel  monitor  the  rssoluticn  of 
all  DPRs  [Ref-  32].  Figure  3.7  [Hef.  32]  is  an 
example   of  the   DPR   used   at    TRM. 

2.  Software  Problem  Raport  (SPR)  -  The  SPR  is  a  reguest 
for  permanent  changes  to  intsrnally  controlled  code. 
It  includes  a  description  of  the  problem,  identifies 
the  library  and  routines  affected  and  proposes  a 
problem  solution.  3  nee  approved  by  the  CCB  it  serves 
as  an  authorization  to  'updat'=  the  master  librarv 
[Ref.  32],  Fiaurs  3.3  [Ref.  32]  is  an  example  ot 
TRW*s    Software   Problem   Report. 

3.  Temporary  Modification  Notice  (TMN)  -  The  TMN 
reguests'and  implements  temporary  changes  to  base- 
lined  code.  Included  with  the  TMN  are  a  listing  of 
actual  changes,  reasons  for  the  change,  any  restric- 
tions, testmq,  verification  and  files  affected.  The 
TKNs  are  correlated  to  SPRs  which  implement  the 
permanent  change  [Ref.  32].  Figure  3.9  [Ref.  32]  is 
an   example  of    a   Temporary    Modification    Notice. 

i.      Benefits 

Kurt      F.      Fischer      notas    in      [Ref.    30]   that      TRW 

received   the      following   benefits   through      the   implementation 

of   its   QA   Program: 

1.  It  has  provided  increased  management  visibilty  into 
the   development   process    through    reviews   and   audits. 

Project  risk  has  been  reduced  through  better  require- 
ments  traceability   and   more      disciplined   and  thorough 

testing. 

It   has   enforced  software   standards. 
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Figure  3.7        Design   Problem   Report. 
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U.  The  development  and  maintenance  of  software  zcols^  has 
been   centralized, 

5.  Quality  Assurance  records  have  been  centralized. 
These  include,  oroblem  reports,  leviations  and 
waivers,  reviews  and  audirs,  'and  -est  and  inspection 
reports   aDoong   others. 

6.  A  skill  center  for  personnel  with  multi-project  visi- 
bility whc  are  better  able  to  prepare  project  plans 
and    procedures. 

7.  An  independent  group  assuring  that  deliverable  items 
meet   contractual   re  quiremenrs. 

j.      Lessens    Learned 

Kurt  Fischer  states,  "Probably  the  largest 
lesson  learned  is  that  one  key  to  the  successful  development 
of  software  is  the  employment  of  a  strong  QA  activity", 
[Ref-  30].  In  addition  tD  the  abov=,  Fischer  points  out  in 
[Sef.  30]  that  the  following  lessons  were  learned  by  THW 
during  the    implementation    of    its    QA    Program: 

1.  Insure  adequate  QA  participation  durina  the  proposal 
and   contract    definition    phases. 

2.  Hiring  personnel  knowledgeable  in  software  and  then 
training  them  in  quality  assurance  is  easier  than 
hiring  QA    people   and    training    them    in    software. 

3.  Perform  the  first  audit  early  in  the  development 
proc«=ssto    allow   plenty   of    tin?    for    corrective    action. 


4. 


Announce  the  audit  well  before  it  takes  place.  The 
object  is  to  ensure  it  was  done  riaht,  net  to  find 
problems. 


5.  Construct  an  audit  checklist  and  distribute  it  to  the 
area  being  audited  at  the  same  time  the  audit  is 
announced.  This  eliminates  subjective  assessments  by 
the  auditor  and  informs  the  party  being  audited  of 
the  exact  scope  and  depth 

of    the   audit. 

6.  Assian  QA  engineers  on  along  term  basis  so  that  they 
may   develop    a    relationship 

with  each  project  sub-aroup.  QA  engineers  should 
cclocate    with   the    development 

personnel  so  that  they  might  better  understand  the 
problems    of    other    project    members. 
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Figure  3.8        Software   Problem  Report- 
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2.      H^vai   Ocean    Systems    Center 
a.      Background 

The    Naval      Ocaan   Systems      Center    (NOSC)  estab- 

lished its  software  Quality  Assurance  program  to  assist 
procuring  activities  in  acquiring  quality  software.  Quality 
software  is  defined  as  software  which  meets  all  requirements 
of  operability,  reliability,  and  maintainability.  The 
expressed  mission  of  the  Software  Quality  Control  organiza- 
tion is  to  provide  assistance  to  project  managers  in  the 
dcquisit ion/mana gemenr  of  higher  quality  software  products 
thrcuch  "'"he  IiudI -^inent aticr.  c^  c-rtdir  ~~3.r.d!5.r'i  oraCticrS.  T~ 
provides  a  manageable  structure  to  the  software  development 
process  through  document  inspection,  configuration  manage- 
ment, and  testing  support.  Spaniard  techniques  include 
inspecting  and  evaluating  "^he  documentation  of  computer 
sysrems,  evaluating  a  projects  configuration  management 
procedures  in  regards  to  software  document arion  and  computer 
programs,  assuring  the  integrixy  of  tested  programs  through 
program  library  control,  increase  user  confidence  through 
testing,  and  being  an  acrive  parrioipant  on  project  Change 
Control    Boards. 

The  activities  of  the  Software  Quality  Control 
organization  are  geared  to  the  projects  life-cycle.  This  is 
true  whether  the  project  is  short,  requiring  a  minimum 
effort,  or  long,  extending  over  a  multi-year  period.  Because 
of  this  fact  each  SQC  st2 p  is  tiei  to  the  project  plan, 
milestone  schedule,  and  list  of  configuration  items  expected 
to    be   baselined    [Ref.    33]. 

At  NCSC  the  Quality  Assurance  office  has  been 
established  at  the  directorate  level.  Figure  3.10  [Ref.  33] 
shows    the   NOSC   organizational   structure. 
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Figure  3.9   Tamporary  Modification  Notice. 
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Figure   3.10        NOSC    Organizational   Structure. 

b.      Quality    Assurance   Dbjectives    and   Policies 

The  following  are  the  most  significant  of  *he 
objectives  established  for  the  Software  Quality  Assuranc 
organization: 

1.  Ensure   consistency   of  software    baseline  development. 

2.  Ensure   compliance   with   design    standards. 

3.  Ensure   compliance   with   programming   standards. 
U.      Ensure   adequacy  and    completeness    of   testing. 
5.      Review   all  test   results.       [Ref.    33] 
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The  folicwing  policies  have  been  established  at 
NOSC    in    regards    to   Software    Quality    Assurance: 

1.  The  develcpinq  iirBCtorate  his  the  basic  r^^soonsi- 
bility  for  the  qualitv  of  products  delivered  by" NOSC. 
Each  Director  shall  utilize  the  established  quality 
assurance  resources,  as  appropriate,  to  assure 
adequate    quality    of    all  end    products. 

2.  The  Director  of  the  Engineering  and  Computer  Sciences 
directorate  acts  as  Center  managements  agent  for 
product  quality  assurance  on  all  Center  projects.  As 
such  he  will  keep  Center  raanagement  infDrmed  of  the 
results  of  reviews  and  audits' cf  products  developed 
and   produced    by  the    center. 

3.  The  Quality  Assurance  office,  which  reports  directly 
to  -he  Director  of  the  Snqineering  and  Computer 
Sciences  directorate,  will  be  the  point  of  contact 
for  the  coordination  of  all  Center  qualitv  assurance 
?.ctivities.  The  QA  office  is  responsible  for  keeping 
"^-he  Center's  QA  policies  current  and  iresponsi'/^  zo 
higher    Navy    directives   and   ins  true-ions.      [  Ref .    33] 

c.  Quality    Assurance  Planning 

The  Software  Quality  Control  Plan  is  prepared  by 
SQC  personnel  through  interviews  conducted  with  the  Project 
Manager.  The  plan  is  coordinated  wi-^h  project  plans  and 
provides  a  description  of  how  ths  elements  of  quality 
management  will  be  appliei  to  the  pr:?ject.  Each  SQC  plan  is 
tailored   -o   project    requirements. 

For  the  Quality  Assurance  program  to  be  effec- 
tive the  SQC  plan  must  contain  certain  elements.  These  are: 
planning  for  products  at  the  end  of  each  task  using  manage- 
ment audits  and  reviews  as  a  measure  of  completion  of  the 
products,  developing  documentation  in  the  proper  sequence, 
and  accepting  SQC  inspection  assistance  to  the  Project 
Manager  as  an  aid  in  ensuring  successful  system  turnover 
[Ref.    33]. 

d.  Software   Standards 

Software  Quality  Control  personnel  are  respon- 
sible for  developing  and  enforcing  design  and  programming 
standards.      In   the      area    of    design,      standards      dealing   with 
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clarity,  detail,  logical  sfficiency,  tschnical  caaturi-y,  and 
consistency,  with   functional    specifications    must   be   enforced. 

Programming  standards  provide  the  required 
consistency  in  the  techaique  and  processing  required  for 
continued  software  support  throughout  the  system's  life- 
cycle.  The  programming  standards  deal  with  nhe  following 
areas:  logic  and  coding  conventions,  flow  chart  standards, 
intermodule  communications,  programming  language  structure 
and  use,  data  design,  module  segmentation,  and  logic  error 
checking. 

Quality  Assurance  personnel  review  the  program- 
ming effort  with  regard  for  compliance  wirh  standards.  If 
any  non-compliance  is  found  they  will  take  steps  to  ensure 
conformity    with    *he    standards. 

e.      Quality    Assurance   Reviews    and    Audits 

The  software  review  process  exists  so  that  a 
qualified  decision  may  be  made  to  recommend  advancement  from 
one  phase  to  the  next.  Audits  on  the  other  hand  are 
conducted  to  verify  configuration  items  conform  to  specifi- 
cations and  other  contract  requirements.  The  results  of 
reviews  and  audits  are  reported  dir=ctly  to  the  development 
directorate   by  quality   assurance    personnel. 

In  addition  to  the  reviews  and  audits  the 
process  of  baselining  will  be  discussed.  It  is  included  here 
because  it  is  an  integral  part  of  the  audit  and  review 
process. 

Baselines  are  a  configuration  management  tech- 
nique used  to  control  the  development  of  a  software  product. 
as    stated   in  fRef.    3U], 


A  projects  software  configuration  is  the  prevailing 
state  of  its  softwar  a  components.  Those  components 
which  are  subjected  to  systematic  management  are  termed 
Configuration  Items  (CIsj  .  Software  CIs  are  qualified  as 
being  elements  of  an  evolving  software  product  which  are 
set        forth         in      technical        docuaentaticn         (including 
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specifications,   drawings,    and    listings),    and    achieved 
the    computer   proaram   i-sslf    (resid^n-    on    card,    taps, 
disk),  " 

In  [Ref.    33]   it    goes    on   to    point  out. 


Baselines  are  employed  throughout  the  life-cycle  of  a 
configuration  item  to  ensure  orderly  transition  from  one 
najor  commitment  point  to  the  next  in  the  system  engi- 
neering, software  development  production,  and  logistic 
support  processes.  Baselines  are  established  at  those 
points  m  a  program  wha re  it  is  necessary  to  define 
fcrmal  departure  points  for  control  of  future  changes  in 
performance,  design,  development,  production,  and 
related   tecnnical    requirements. 


'"h^    base'^i*^—      is   c~^^'*"ed      upon    sccso^arce      of   a      develop m'-'nt 

document    or   prodix:t. 

There  are   four    types   of      baselines   used    at    NOSC, 

Functional,    Allocated,      Product,      and    Operational    [Ref.    34]. 

A   brief    description    of   each    follows. 

1.  Functional  Baselina  -  This  considered  the  highest 
level  baseline.  The  technical  documentation  at'this 
level  delineates  all  the  nec=ssary  functional  charac- 
teristics,- the  t2S-^s  which  will  be  required  to 
demonstrate  their  achievement,  all  necsssary  inter- 
facesj  and  any  and  all  design  constraints.  This 
baseline  generally  covers  a.il  the  documentation 
produced  pfor  to  davelopment  of  the  software  design 
and   the    formal   System   Design    Review. 


2.  Allocated  Baseline  -  At  this  the  next  lower  level 
baseline,  the  performance  oriented  specifications, 
which  are  subordinate  to  the  Configuration  Items  of 
the  functional  level,  expand  upon  allocated  func- 
tional characteristics.  All  documentation  produced 
short  of  the  Preliminary  Design  Review  is  covered  by 
this   baseline. 

3.  Product  Baseline  -  This  is  considered  the  lowest 
level.  The  docuinea tation  at  "rhis  point,  which  is 
subordinate  to  both  the  functional  and  allocated 
levels,  defines  the  production,  operation,  mainte- 
nance, and  logistic  support  phases  of  its  life-cycle. 
This  normally  covers  all  documentation  and  programs 
produced    prior  to   the   Formal    Qualification    Review, 

U,  Operational  Baseline  -  One?  the  developed  system 
passes  the  Fcrmal  Qualification  Review  and  has  proved 
it  meets  operational  requirements,  the  operational 
baseline  is  established.  All  modifications  required 
to  the  system  during  its  life-cycle  will  be  performed 
from  this  baseline. 
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The      baselining    techr.iqas      is   also      used    by      TRW    during      its 

software   development    procsss. 

Following   is    i    list    of      the    reviews    conducted   by 

the    Software   Quality   Control   organization      at   NOSC,       A   brief 

description   of   each    review    is   included. 

1.  Initiation  Review  -  The  purpose  of  this  review  is  to 
affirm  the  Operational  Requirement  as  the  basic 
Quideline   for   the      project.      Prior    to   the      review   the 


.___^,y        __^ ,  _--         _ 

requirement  is  reviewed  to  ensure  no  requirements  or 
constraints    have   been   omitted   [Rsf,    33]. 

2.  Systems  Requirement  Review  -  The  objective  of  this 
review  is  to  determine  the  adequacy  of  the  develop- 
er's efforts  in  dsfininq  system  requirements.  The 
review  is  conducted  once  a  signif icant  portior.  oz  --a  = 
system  functional  requirements  have  been  established 
[ief.    33]. 

3.  Document  Feview  and  Substantiation  -  This  occurs  in 
each  Dhas€  of  the  iavelopment  life-cycle.  The  primary 
conc<=fn  is  to  review  draft  documentation  for 
completeness  and  correctness.  Quality  assurance 
personnel  inspect  and  substantiate  the  contents  of 
all  documentation.  The  documentation  under  review  is 
compared  against  established  standards  to  ensure  it 
contains  the  proper  level  of  detail  and  that  all  the 
required  content  is  present,  Not  only  is  a  single 
oiece  of  documentation  reviewed  by  itself  but  it  is 
compared  to  all  associated  documentation  to  ensure 
comoleteness  and  consistency.  All  deviations  from 
standards  and  any  technical  problems,  are  noted  and 
submitted  to  both  the  Project  :ianaaer  and  the  devel- 
oper for  correction.  Once  the  Operational  Requirement 
has  been  reviewed  and  approve!,  the  remainder  of  the 
documentation  produced  by  the  project  is  reviewed  to 
ensure  it  is  consistent  with  and  meets  the  require- 
ments set  forth  in  the  Operational  Requirement 
[Ref.    33]. 

fx.  System  Design  Review  -  Once  the  project  team  has 
determined  that  the  software  requirements  documents 
fulfill  all  requirements  and  present  a  suitable  allo- 
cation of  performance  requirements  between  hardware, 
software,  and  human  actions,  the  System  Design  Review 
is  held.  Also  the  identification,  correlation, 
completeness,  and  risk  of  the  software  requirements 
documents  is  evaluated.  Upon  approval  these  documents 
are   considered   baselined   [Ref.    33]. 

5.      Preliminary      Desian      Review      (PDR)  -      The      primarv 

concerns  of  the  ?DR  are  the  software  design  and  the 
completion  of  requirements  set  forth  in  previously 
baselined  documents.  The  iteas  which  are  considered 
from   the    program    design   documents    include: 

a)  Computer    program    functional    flow   charts. 

b)  Storage  allocation   charts. 

c)  Control  functional   descriptions 
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d)  Structure   and   organization    of    the    data    base. 

e)  Functional    interfaces.      [Ref.    33] 

6.  Critical  Essian  Reviews  -  These  reviews  are  conducted 
as  individual"  system  programs  or  modules  are  speci- 
fied. The  primary  concerns  hare  are  that  required 
standards  are  met,  all  prior  baselined  functions  are 
fulfilled,  and  that  prior  to  coding,  the  lowest  level 
of  design  detail  has  been  reached.  The  following 
items  [aef.  33]  are  under  consideration  from  the 
program   specifications: 

a)  Compat ability  Df  design  with  functional  inter- 
faces. 

b)  Data   base    interactions. 

c)  Design  integrity  of  logic  diagrams,  algorithms, 
storage  allocations,    and    flow    charts. 

d)  H .-ir dw are    ir.terf3.ces. 

e)  Human    interfaces. 

7.  Formal  Qualification  Review  -  This  is  conducted  upon 
complexion  of  both  the  Func-cional  Configuration  Audit 
and" the  Physical  Configuration  Audit  and  after  all 
discrepancies  uncovered  by  these  audits  are 
corrected.  The  Djrpose  here  is  to  ensure  that  the 
system  as  developed  and  testel  meets  all  requirements 
set  forrh  in  the  Operational  Requirements.  From  this 
review  it  is  determined  that: 

a)  User  requirements  have  beea  satisfied. 

b)  Documentation  is  sufficisnt  to  support  the  system 
throughout  its  life-cycla. 

c)  User  functions  are  aiequataly  described  and  docu- 
mented. 

d)  Testing  is  sufficient  to  ensure  user  confidence. 

e)  All  outstanding  deficiency  reports  have  been 
resolved. 

f)  The  outcome  of  the  review  will  either  be  a 
successful  comoletion  or  the  determination  that 
further  development  is  necessary  [Ref-  33]- 

8.  Post  Development  Reviews  -  These  reviews  are 
conducted  as  the  operational  evaluations  of  the 
system  are  made.  The" focus  is  on  system  development 
problem  areas,  operational  iif f iculties,  and  unde- 
tected deficiencies  [Ref.  33]. 

The   following  list   contains   the  major   audits 

conducted  by   the   NOSC  Software   Quality   Assurance   group 

during  the  software   development  process.    As  was   the  case 

with  the  reviews,.   a  briaf  description  of   each  audit   is 

included. 
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1.  Ir.-Process  Configuration  Audit  -  This  serves  as  the 
orimary  ireans  or  valiiiting  that  development  of  a 
software    Configuration   I  tern    nas    been   comple-ced    sa-is- 


ciencies  is  maintained.  The  updated  document  and  all 
deficiency  reports  are  inputs  to  this  audit 
[Ref .    33  ]• 

2.  Functional  Configuration  Audit  -  This  is  a  critical 
comparison  of  an  item's  test/analysis  data  and  func- 
tional specifications  to  validate  that  the  item  as 
designed  and  developed,  meets  all  the  functional 
performance  reguirem ents  specified  in  its  development 
specification   [Ref-     35]- 

3.  Physical  Configuration  Audit  -  This  is  a  comparison 
of  the  "as  bu2.it"  item  with  its  approved  and  released 
technical  documentation.  The  objective  is  to  ensure 
rhat  the  documentation  is  complete  and  is  appropriate 
for  operational  maintenance"  and  support  purposes 
[Ref.    35]. 

Both   the    Functional   Configuration      Audit   and  the 

Physical     Configuration   Audit      are      conducted      prior   to      the 

submission   of      a   configuration      item    for      formal   acceptance. 

From   [Ref.    33]   the    purposa    of   these    audits    is: 

1.  Confirm  compliance  to  change  control  procedures  and 
to  ensure  only  approved  changes  have  been  imple- 
mented. 

2.  To    ensure   that   objectives   are    sufficient, 

3.  To  ensure  the  develDped  software  product  is  the  same 
as   the   specified    software    product." 

f.      Testing 

Software  test  and  evaluation,  as  conducted  by 
the  Software  Quality  Assurance  personnel,  is  designed  to 
ensure  that  the  software,  as  developed,  meets  the  original 
reguirements  set  forth  by  the  user/sponsor  in  the 
Operational  Reguirement  and  that  it  performs  as  defined  in 
the  system  documentation.  By  conducting  an  analysis,  tech- 
nical evaluation,.  and  a  detailed  review  of  project 
documentation  the  necessary  inputs  to  prepare  test  plans, 
test  specifications,  and  test  procedures,  are  obtained. 
These  documents  are  then  used  in  the  actual  testing  and 
evaluation  of  test  results  to  verify  that  the  system,  as 
developed   meets    all    technical   specifications  [Ref.    33]. 
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Following  is  a  list  of  tha  various 
and  activities.  A  brief  discussion  of  9ach  is  included. 

1.  Test  Plan  -  This  is  written  by  Software  Quality 
Control  personnel  to    definr  the 

scope  or  tests  required  to  assure  thar  the  software 
meets  all  required  specifications.  Although  the  test 
plan  is  a  high  levsl  documsnt,  from  which  the  -est 
specifications  are  written^  it  still  musx  identify 
the  degree  of  testing  and  ths  specific  functions  to 
be  tested.  The  schedule  for  individual  tests  and  a 
summary  of  the  snvironmear  to  be  used  are  also 
included  in  the  iiest  plan.  The  plan  is  reviewed  by 
the  software  developers  and  approved  by  the 
Program/Project  Manager  [  Ref .  36]. 

2.  Test  Specifications  -  Software  Quality  Control 
prepares  a  test  specification  for  each  test  m  the 
test  plan.  Based  on  re quirenents  set  forth  in  the 
design  do cumentat ion  _  of  the  developing  system,  the 
tes*  specif icatiou  defines  ths  basic  ":~s-  cri'c^ria 
and  the  general  methods  to  be  used  in  a  specific 
test.  Once  a  test  specification  is  prepared  i-  forms 
the  basis  for  the  development  of  test,  procedures.  In 
addition  to  defining  the  scop=  of  the  specific  tests^ 
test  specifications  state  the  purpose  or  the  test  ana 
identify  the  software,  hardware,  and/or  system  to  b« 
tested.  There  must  be  sufficient  information  in  a 
test  specification  so  that  test  procedures  may  be 
developed  and  so  the  results  of  defined  tests  may  be 
evaluated.  These  are  also  reviewed  bv  the  software 
developers  and  approved  by  the  $>rogram/Pro  ject 
Dianager  [  Pef .  36].  ' 

3.  Test  Procedures  -  These  procedures  are  developed  by 
the  Software  Quality  Control  personnel  using'  test 
specif ica-.ions^  users  manuals,'  and  other  relsvan*: 
design  documentation.  The  prime  purpose  of  the  test 
procedures  is  to  present  detailed  instructions  for 
both  test  execution  and  the  evaluation  of  test 
results.  The  organization  and  structure  of  the 
processes  are  expressed  in  general  terms  in  the  test 
procedure  along  with  constraints  or  assumptions 
imposed  on  their  asage.  A  description  of  the  total 
equipment,  manpower,  computer  program,  and  supporting 
documentation  required  for  operation  is  also 
provided.  All  hardware  or  software  revisions  or  modi- 
fications must  be  specified  along  with  any  required 
ore-test  checkout  required  to  ensure  a  valid  test 
environment  [Ref.  36]. 

U.  Software  Testing  -  Software  Quality  Control  personnel 
may  perform  testing  in  either  of  two  environments: 
simulated  or  "on-eite".  In  a  simulated  environment 
certain  subsystems,  links,  and  peripherals  are  avai- 
lable for  the  express  purpose  of  production  and 
testing.  "On-site"  testing  is  done  using  the  system 
in  real-life  environment.  The  oolicies  and  procedures 
are  set  forth  in  the  test  plan,  specifications,  and 
procedur esused  in  accomplishma  "  software  testing 
tRef.  36]. 

5.  Test  Reports  -  As  each  test  is  completed  a  Test 
Report  IS  written  to  document  the  satisfactory  or 
unsatisfactory  completion  of  the  test.  Any  and  all 
deviations  from  test  procedures  or  eauipment;  malfunc- 
tions must  also  appear  on  the  Test  Report.  Each 
apparent   system  discrepancy   will   be   noted  by   the 
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submission  of  a  ssperate  incident  report.  The  Test 
Rf^pct  V i  il  zreferei  ce  2.12.  cojuoleted  test  oTcced'izre 
s-teps  to  allow  programmers  to""  duDlicate  the  condi- 
tions in  which  any  apparent  incident:  was  discovered. 
All  completed  Tesc  Reports  become  a  parr  of  the 
permanent  system  do::umentatioQ  [Ref,  36]. 

g.   Problem  Reporting  and  Reviews 

As  was  the  case  with  IRW  the  area  of  change 
control  is  also  discussed  in  this  subsection.  The 
Configuration  Control  Board  and  the  documents  used  in 
configuration  control  are  discussed  in  the  list  that 
follows. 

1.  Ccnf iaur ation  Change  Control  -  The  purrcse  of 
configuration  chancie  control  is  to  manage  and' monitor 
changes  or  modifications  to  baselined  configuration 
items.  The  sponsor,  user,  software  developer,  or  any 
other  member  of  the  orojec^  organization  may  oropose 
changes  to  the  software.  These,  along  with  any  prob- 
lems uncovered  daring  test  and  evaluation,  are 
presented  to  the  Configuration  Control  Board  (CCB)  as 
either  Engineering  Change  Proposals,  deviations.  or 
waivers.  The  CCB  in  turn  investigates  all  cnange 
requests,  deviations,  and  waivers  and  based  on  docu- 
mented analysis,  recommends  proposed  action 
fRef.  33].  The  CCB  is  mad?  up  of  representatives 
from  all  project  affected  activities.  "  The  Project 
Manaaer  is  tne  chairman  and  the  Quality  Assurance 
representative  acts  as  recoriina  secretary.  Althouqh 
■^he  final  decision  regardina  all  changes  ultimately 
rests  with  the  chairman,  he  solicits  expert  advice 
from  project  participants  such  as"  Software 
Development,  Systems  Engineerina,  Quality  Control, 
Logistics  Support,  Fleet  User,  and 
Facilities/Hardware  management  [ Ref .  3U]. 

2.  Enqineering  Change  Proposal  -  Used  in  submitting 
proposed  cnanges  to  basexined  software  conf iauration. 
Either  DD  Form  1692  or  1693  is  used  [Ref.  34]. 

3.  Request  for  Deviation  -  This  is  used  when  it  is 
necessary  to  depart  temporarily  from  documented 
requirements.  DD  Form  1694  is  used  in  this  case 
[Ref.  34]. 

4.  Request  for  Waiver  -  If  an  item  fai^s  to  conform  to 
Its  required  conrig uration  and  this  is  due  to  a 
development  error,  a  Request  for  Waiver  is  submitted. 
This  is  also  submitted  on  a  DD  Form  1694  [Ref.  34]. 

5.  Engineering  Change  Orders  -  Once  approval  ^^.s  been 
granted.  Engineering  Change  Proposals,  Deviations, 
and  Waivers  are  implemented  throuah  an  Engineering 
Change  Order  [Ref.  3  4]. 

6.  Specification  Change  Notice/Notice  of  Revision  -  Both 
these  documents  are  used  when  an  Engineering  Change 
Proposal  or  Waiver  affects  baselined  documents  or 
drawings.  DD  Form  1696  is  used  for  the  SCN  and  DD 
Form  1695  is  used  for  the  NOR  [Ref.  34]. 
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7.  Software  Trouble/Incident  Report  -  This  is  used  for 
reporting  3.11  deviiticns  froiat^st.  Procedures,  ecjuio- 
irient  tnalf  unctions  and  software  anamolies  [Ref-'SS"]. 
These  become  a  part  of  the  software  Test  Repori  and  a 
permanenr    part   of    sy srem    docunentation. 

h.      Program    Library    Control 

Program  Library  Control  is  designed  to  maintain 
and  control  a  systems  computer  programs  and  all  changes  to 
these  programs.  Software  Quality  Assurance  personnel  are 
responsible  for  approving  all  changes  to  library  programs. 
This  system  not  only  provides  baseline,  error-free,  patch 
free  programs  fcr  test  and  evaluation  but  it  also  insures 
that  all  approved  changes  are  incorporated  into  the  programs 
[Ref.    37].      TRW    uses    a  system  similar    to   this. 

i.      Benefits 

In  [  Hef .  33]  it  states  that  NOSC  has  realized 
the  following  benefits  through  the  establishment  of  its 
Software  Quality   Assurance    program: 

1.  It  establishes  a  controllable  structure  in  the  soft- 
ware  development    process. 

2.  It  assures  certain  elements  of  quality  in  every  phase 
of   the   development. 

3.  3y  reducing  rework  s  ubst  antiallv ,  it  orovides  for  th^* 
satisfaction    of   requirements. 

U.  The  substantia!],  i^eduction  in  the  amount  of  rework  has 
lead  to   a   significant   savings    m    life-cycle   costs. 

3-      General    Electric   Com^an^ 

a.      Introduction 

Tha  software  quality  assurance  activities  at  two 
separate  divisions  of  the  General  Electric  Company  will  be 
discussed.  One  discussion  will  cover  the  Electronic  Systems 
Division  located  in  Syracuse,  New  York.  The  other  will  cover 
the  Space  Division  located  in  King  of  Prussia,  Pennsylvania. 
Neither    discussion    will      b3    extensive    as    both      divisions    use 
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quality    assurance      techniques   which    are      similar   to      rh?   two 
organizations    previously    iascribed. 

b-      Electronic  Systems  Division 

(1)  .  Quality  A§§li£§iica  Objectives.  John 
McKissick  Jr.  and  Robert  k.  Price  point  out  in  [ Ref .  38]  the 
objective  of  the  Computer  Software  Quality  Assurance  (CSQA) 
Program  at  the  G.  E.  Electronic  Systems  Division  is  to 
ensure  that  software  delivered  under  a  contract  meets  the 
requirements  of  the  contract.  As  can  be  seen  this  is  similar 
to   the  organizations   previously   discussed. 

(2)  .  I2ii2.1it;i  ^ssurancr  ?  lannir.a.  The  manager 
of  Computer  Software  Reliability  and  Quality  Assurance  and 
dedicated  technical  specialists  are  responsible  for  planning 
and  implementing  the  QA  Program.  In  addition  to  providing 
s-aff  support  to  project  managers  the  manager  of  Computer 
Software  Reliability  and  Quality  Assurance  reports  directly 
to  the  manager  of  Reliability  and  Quality  Assurance.  It 
should  be  noted  that  the  Zoraputsr  Software  Reliability  and 
Qualit-y  Assurance  group  is  organizationally  independent  of 
Software    Engineering. 

After  reviewing  both  the  Computer  Program 
Management  Plan  and  the  Software  Standards  and  Procedures 
Manual,  Quality  Assurance  personnel  develop  the  Computer 
Software  Quality  Assurance  Plan  [Raf.  38].  The  plan  defines 
all  activities  which  control  and  assure  computer  software 
quality.  The  plan  is  developed  usiag  Mil-S-52779A ,  and  it 
identifies  the  organizational  component  responsible  for  each 
activity. 

(3) .  Software  Standards.  Similar  to  TRW  the  G. 
E,  Electronic  Systems  Division  also  develops  a  Software 
Standards  and  Procedures  Manual.  This  document,  which  is 
primarily  used  by  the  programming  teams,  establishes  rules, 
guidelines,      and      limitations  which      are   to      be   observed      in 
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generating  so f-^  ware  designs  and  croia  which  will  have  *he 
properties  of:  consistancy,  readabiii+y,  and  quality 
[Hsf,    38]-  The      structure      irA    content      of      the      Software 

Development   Notebook    (SDN)     is  also   defined. 

The  SDN  is  similar  to  the  Unit  Development 
Folder  used  by  T HW.  It  is  a  simple  loose-leaf  notebook  which 
is  established  during  the  Preliminary  Design  phase  for  each 
Computer  Program  Component  (module) .  It  provides  a  common 
collection  point  for  all  information  dealing  with  a  CPC  and 
it  is  maintained  and  updated  throughout  the  remaining  phases 
of   the   software    development    [Ref,    38]- 

The  SDN  is  broken  down  into  the  following 
sections:  Requirements,  Detail         Design,  Functional 

Capabilities,  Code,  Test  Case  Dsscriptions,  Test  Case 
Results,  Software  Problem  Reports,  and  ?1iscellaneous 
Information.  With  the  exception  of  Software  Problem  Reports 
and  Miscellaneous  Information,  each  section  contains  a  cover 
sheet  shewing  schedule  datas,  actual  completion  dates,  and 
review   and    approval    signatures   [Ref.    38]. 

The    SDN,  like    IRW's    uDr,       serves      as    the 

principle  working  document  of  the  programmer.  In  addition  it 
provides  management,  the  customer,  and  QA  personnel  visi- 
bility into  the  design,  status,  and  quality  of  the  software 
under    development  [Ref.    38]. 

SDNs  are  audited  monthly.  This  audit  may 
be   on   either   an    announced   or   unannounced    basis. 

(4)  .  Quality  M§lilsli2e  Reviews  and  Audits, 
Audits  and  reviews  similar  to  thos=  conducted  by  both  TRW 
and  NOSC  are  conducted  at  the  G.  E,  Electronic  Systems 
Division.  Internal  reviews. are  conducted  by  software  and 
systems  engineers  who  have  not  contributed  to  the  design 
under  review.  The  review  chairman  is  responsible  for  imple- 
menting the  review  plan  which  was  approved  by  Quality 
Assurance   and  the  Project   Manager. 


89 


Aft<=ii:  th=  ir.ternal  reviews  are  ccr.duc~ed, 
joir.r  cust.omer/c  cnxracror  reviews  arrr  held.  These  reviews 
provide  a  technical  foram  for  better  inut'ial  andars*aniing  of 
tha  performance  requirements  allocared  to  the  computer  soft- 
ware, and  of  the  design  approach  selected  to  meet  these 
requirements  [Ref.    38  ]• 

(5) .  Testing.  At  the  G.  E.  Electronic  Systems 
Division  the  Test  Plan  is  developed  during  the  preliminary 
design   phase   of    the    development    process.    Test    procedures 

are  developed  durinq  the  detailed  design  phase.  Actual 
testing  occurs  in  the  final  four  phases  of  the  development 
process:  Cede,  Debug  and  'Jr.ir  Test:,  Deveiccinent  Testing, 
Integration   Testing,    and    Acceptance    Testing   [Ref.    38]. 

Onit  testing  is  conducted  on  individual 
routines  to  reveal  coding  errors,  computational  errors, 
improper  input  handling,  inappropriate  error  messages,  and 
incorrect:  formatting  and  content  of  output.  Development 
testing  takes  the  prvioiisly  tested  routines  and  combines 
them  to  form  Coccput^r  Program  Components  (C?C)  .  The  func- 
tional capabilities  of  the  CPC  is  are  then  verified. 
Integration  testing,  which  is  performed  by  an  independent 
test  team,  combines  CPCs  and  verifies  the  correct  sequencing 
of  components,  compatible  component  interfaces,  and  proper 
data  routing.  Acceptance  testing,  which  is  also  done  by  an 
independent  team,  verifies  system  level  functional  require- 
ments. These  include  overall  timing  and  the  ability  to 
handle  the  total  input  load.  As  at  TRW  and  NOSC  QA  personnel 
witness   all   testing    [Ref.    3  8]. 

(6)  •  Problem  Re^ortin^  and  Review.  LiJce  both 
TRW  and  NOSC,  G.  E.  Electronic  Systems  Division  has  a 
Configuration  Control  Board  which  reviews  and  approves  all 
changes  to  the  software.  As  in  both  the  previously  described 
organizations.  Quality  Assurance  personnel  are  members  of 
this    board.    Quality    Assurance   personnel    are   also    responsible 
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for  verifying  that   all  approved  changes  have  been  incorpo- 
rated. This  is  similar  to  both  NOSC  ^nd  TRW. 

A  Software  Problem  Report  (S?R)  defines 
and  documents  a  problem  and  the  test  conditions  under  which 
it  occured.  Quality  Assurance  receives  a  copy  of  all  SPRs. 
In  addition  they  maintain  a  listing  of  all  outstanding  SPRs 
and  the  party  responsible  for  corrective  action.  Once  a 
problem  has  been  corrected  Quality  Assurance  annotates  its 
copy  of  the  SPR  with  the  way  in  which  the  problem  was 
corrected  [  Ref-  38]. 

All  coding  changes  are  authorized  using  a 
Software  Change  Order  (SCO)  .  Changes  to  hardware  and  soft- 
ware specifications  are  documented  on  a  Specification  Change 
Notice  (SCN)  [Ref-  38]. 

(7) .  Benefits.  An  effective  Quality  Assurance 
Program  allows  rhe  G.  E.  Electronic  Systems  Division  to 
deliver  computer  software  which  meets  all  contractual 
reguiremen-iis.  In  addition  it  provides  management  visibility 
into  the  software  development  process. 

c.   Space  Division 

(1)  .  Quality  Assurance  Objectives.  The  Quality 
Assurance  Progratr  which  was  reviewed  has  been  in  effect  at 
the  General  Electric  Company's  Space  Division  sine  e  1978. 
The  primary  objective  of  the  program  is  to  ensure  that. 
delivered  software  meets  all  contractual  requirements.  A 
secondary  objective,  which  is  designed  ro  help  secure 
contracts,  is  to  define  and  implement  specific  measures 
designed  to  ensure  delivered  software  incorporates  the 
features  necessary  to  achieve  testability,  maintainability, 
reliability,  etc.   [Ref.  29]. 

(2) .  Quality  Assurance  Planning.  As  is  -he 
case  in  the  three  previoasly  described  organizations  the 
primary  job  of  the  Qualit7  Assurance  group  is  to  prepare  -he 
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QA  Plan.  Once  the  QA  Plan  has  been  developed  Quality 
Assurance  rums  its  attention  to  the  implement  a- ion  and 
managemer-t   of    the  plan. 

(3) .      Software         Standards.  Each         programmer 

working  on  a  given  project  ricaives  a  copy  of  the 
Programming  Standards  Docament  (PSD),  which  outlines  the 
standards  to  be  used  to  produce  3.  high-quality  software 
product.  It  is  not  enough  however  that  the  programmer  be 
given  the  PSD  and  left  to  go  along  h.Ls  merry  way.  Training 
sessions,  conducted  by  the  Software  Development  group,  are 
held  to  explain  the  conteat  of  the  PSD.  It  is  the  job  of  the 
Quality  Assurance  group  zo  verify  that  each  programmer 
participates   in    the    training  sessions    [Ref.    29]. 

(U)  .  Quality.  Assurance  Reviews  and  Audits.  The 
G.  E.  Spaci=  Division  conducts  reviews  and  audits  similar  to 
those  of  the  previous  three  organizations.  They  do,  however, 
give  special  attention  to  the  developmentof  software  inter- 
faces. The  development  of  -^.he  software  interfaces  is  done 
by  the  Software  Development  Group.  However  the  System 
Engineering  group  is  responsible  for  developing  the 
Interface  Control  Documents  (ICD)  .  The  purpose  of  this 
process  is  to  provide  a  timely  and  complete  definition  of 
interface  details  and  to  provide  a  continuous  in-depth 
review   process   [Ref.    29]- 

(5) .  Testing.  The  Quality  Assurance  role  in 
the  actual  testing  is  minor.  The  bulk  of  the  Quality 
Assurance   groups   work  is    done   prior   to    actual   testing. 

Quality  Assurance  first  identifies  exactly 
what  is  to  be  tested  and  at  the  same  time  develops  a 
detailed  definition  of  the  test  environment.  They  then 
explicitly  define  both  the  test  and  the  evaluation  process, 
especially   success/failure    criteria. 
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During        the      actual        testing,  O'lality 

Assurance  acts  as  a  monitDr  to  snsare  that  the  previously 
defined  procedures  are  being  followei.  They  also  ensure  that 
any   changes   are    documented    correctly    [ Ref-    29]. 

(6)  -  Problem  R  eporting  and  Review.  Any  prob- 
lems uncovered  during  the  testing  process  are  documented 
using  a  Discrepancy  Report  (DR)  .  Once  testing  is  complete  a 
post-test  meeting  is  held  and  the  Discrepancy  Reports  are 
assigned  to  individuals  for  resolutiDn,  The  Software  Quality 
Assurance  group  monitors  all  outstanding  DRs  to  ensure  that 
all    problems    are    corrected. 

The  Discrepancy  Reports  serve  as  a  measure 
of  product  quality.  The  software  3A  group  analyzes  -^he  data 
provided  by  the  DR  and  prepares  a  3ta tis-^.ical  report.  In 
[Ref.  29]  Stephen  L.  Stami,  Manager  of  Productivity  Programs 
at  the  G.  E.  Space  Division,  points  out  that  such  things  as 
the  number  of  DRs,  the  frequency  distribution  of  DRs  by 
type,  the  mean  time  of  closure  (corr=ction) ,  and  the  DR  rate 
as  a  function  of  product  1  ife  can  be  used  by  inanagemer.t  to 
identify    weak   spots    in  the    software    implementation    process. 

(7)  .  Program  Library.  A  system  similar  to  that 
at    NOSC    is    used    at    the   G.    S.    Space    Division. 

(8)  .  2.]i^lill  i§§li£i2;Sx  I22i§«  Automated  soft- 
ware cede  analysis  tools  are  used  as  part  of  the  test 
program  to  uncover  code  in  the  final  product  which  has  never 
been  executed.  If  this  is  the  case  it  is  determined  if  there 
is  a  hole  in  the  test  program,  a  possible  flaw  in  the 
product  design  or  just  some  superfliioas  code  in  the  finished 
product  [Ref.    29  ]. 

(9) .      Lessons   Learned.         The      following    elements 

[Ref.    29]   have  been    found   by  the    G.    3.       Space  Division    to    be 

necessary      for      a      successful        Software      Quality      Assurance 

Program: 

1.      The   Software      Quality    Assurance      Plan   must      have   high 
project   visibility,     it   must   d  =  fine   the   SQA    program    at 


93 


a  level  of  detail  sufficient  to  allow  implementation, 
ar  5.  it  niust  have  th3  cro^ect  and  coni'oany  !iiana'7'^in9n~s 
whole-hearted    support.'' 

2.  The  aoplication  of  special  software  engineering  tech- 
niques by  the  programming  staff  specifically  targeted 
at   increasing   product  quality. 

3.  The  project  organization  must  distribute  the  Software 
Quality  Assurance  Program  responsibility,  placing  SQA 
tasks   where   the  capability   really   exists. 

U.  The  ability  to  leasure  the  effectiveness  of  the  QA 
program  and,  if  possible,  the  quality  of  the  end 
product. 

5.  Software  Quality  Assurance  personnelmust  be  an 
integral    part   of  the    project    team. 


E.      CONCIUSIOH 

Four  separate  Software  Quality  Assurance  organizations 
have  been  discussed  in  this  chapter.  However  they  are 
similar   in   several    ways. 

First  the  Software  Quality  Assurance  groups  at  each  of 
these  organizations  becoras  involved  with  the  project,  early 
in  its  life-cycle.  Each  of  the  Quality  Assurance  groups  is 
involved  in  every  phase  of  the  development  process.  Each  of 
these  organizations  recognizes  the  fact  that  an  effective 
Quality  Assurance  organization  allows  them  to  deliver  a 
quality  software  product  which  meets  all  contractual 
requirements.  They  are  also  in  agreement  on.  the  fact 
Software  Quality  Assurance  saves  money  in  the  development 
process  by  identifying  and  correcting  errors  early  in  the 
process.  They  also  agree  that  an  effective  QA  program 
provides  both  management  (Corporate  and  Project)  and  the 
customer  wi-^h  visibility  into  the  development  process.  The 
view  of  software  as  a  product  is  also  a  similarity  among 
these  organizations.  Finally  all  these  groups  agree  that 
Software  Quality  Assurance  groups  do  not  create  quality  in  a 
project  but  that  it  is  in  fact  part  of  the  job  of  every 
project   member. 
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17.  THE  22ALrTY  APPROACH  AT  FHSO 


A.   STEOCTORE 


Within  nhe  Naval  Matsrial  Command,  the  Fleet  Material 
Support  Office  (FMSO)  is  a  field  activity  sponsored  by  the 
Naval  Supply  Systems  Command.  FMS3  performs  two  major  func- 
tions in  fleet  logistical  support.  The  one  discussed  here 
is  that  of  principal  Navy  Central  Dasign  Agency  (CDA)  for 
automated  supply,  fir.anciil,  aain  tanar.ce- ,  aad  Icgistical 
systems,  a  process  which  consumes  the  majority  of  FMSO 
resources. 

A  general  descriptiori  of  the  Drganizarional  elements 
directly  related  to  the  central  systems  development  process 
is  depicted  in  figure  u.  1  Withi.i  the  organization  the 
Comptroller  Department  (Code  91).  and  ilanagement  Department 
(Code  92)  are  the  two  departments  that  can  be  considered  as 
staff.  The  other  six  CDA  departments  are  production 
oriented  or  support  the  production  effort  and  are  considered 
as  line  organizations  which  are  directly  responsible  for  the 
development  and  maintenance  of  standard  Automated  Data 
Processing  (ADP)  systems  [Ref.  39]. 

A  system  is  considered  to  be  an  organized  set  of  ADP 
hardware,  environmental/application  software,  and  documented 
procedures  designed  to  automate  the  basic  management  and 
operating  processes  for  a  customer  site  or  group  of  customer 
sites  with  common  missiDn  responsibilities.  "Documented 
procedures"  as  used  above  refers  to  the  applicable  ADP 
related  and  non-ADP  related  procedures  established  to 
support  the  hardware  and  software  aspects  of  the  system 
[Ref.  39]- 
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Figure  4.1        Organizational    Structure. 

The  primary  role  of  the  Managiment  Department  is  to 
coordinate  with  and  sappor-  the  efforts  of  the  Cdh 
Production  Departments.  In  this  capacity  the  branch  that  is 
of  mos-  relevance  to  this  paper  is  that  of  the  Oualit'/ 
Control  Branch.  The  CDA  Development:  Process  Model,  figure 
4.2,  reflects  all  of  the  basic  steps  appropriate  -co  ensuring 
that  each  CDA  tasking  received  by  FMSO  is  effectively 
managed  and  results  in  a  liigh  quality  product  being  released 
for    use      by  the    customer.  The   model    covers      all   projects, 

large  and  small,  new  developments  ou  maintenance.  However, 
it  is  anticipated  that  sone  of  the  steps  in  the  model  may 
not  be  applicable  to  all  projects.  Therefore,  an  explicit 
decision  by  the  appropriate  level  of  management  is  required 
in  order  to  exclude  process  steps  determined  not  applicable 
on      a    project.  As   the      model    is      followed   through,         note 

specifically  the  areas  that  deal  with  symbols  equating  to  92 
QC   and  92   QC   Optional  [Ref.    40]. 
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Figure   4.2        Project  Flow   Model. 
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B.       THE    QOALITY     TROCSSS 

As  a  new  requirement  is  received  by  FMSO  a  mechanism  is 
activated  to  ensure  thar  the  output  produced  will  meet  the 
user*  s  expectations.  This  mechanism  at  FMSO  is  called  the 
quality  process,  that  is,  an  attitude  that  extends  from  the 
individual  programmer  all  the  way  through  the  systems  devel- 
opment cycle  up  to  top  [nanagement.  By  design,  it  is  a 
multi-layered  approach  to  achieving  quality  that  begins  with 
the      System   Development      Quality      PrDcess    (SDQP) •  Falling 

under  the  SDQP  are  all  the  separate  requirements  that  will 
eventuallv  produce  a  workina  ADS.  At  each  stag^  of  develop- 
ment, quality  standards  are  imposed  upon  all  personnel  at 
all  levels  within  the  chain  of  comnand  beginning  with  the 
feasibility  study  and  requirement  definition,  proceeding 
through  the  functional  design,  coapuxer  design,  program 
development,  testing,  operations  and  maintenance,  and  ending 
with    the      Prograrr  Trouble      P.eports    (PTH)  .  within    each      of 

these  particular  evolutions  labeled  above  are  subsections 
that    must    be  completed  before   the    process    evolves    further. 

Layered  on  top  of  the  SDQP  are  the  Quality  Control 
Mechanisms.  These  Quality  Control  Jiechanisms  provide  the 
ability  to  ensure  that  a  quality  and  error  free  product  is 
in  fact  produced.  The  msthods  utilized  to  accomplish  this 
are  good  project  management  during  the  requirement  defini- 
tion and  functional  design  stage,  sound  data  base  management 
during  functional  design  and  computer  design,  an  effec^.ive 
verification  process  during  computsr  design  and  program 
development,  proper  validation  procedures  during  program 
development  and  testing,  and  a  satisfactory  prototype/op 
review  during  testing  and  operations/  maintenance  of  a 
system. 
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On  top  of  both  the  SDQ?  =nd  Quality  Control  Mechanisms 
we  have  the  Quality  Production  Cor.cspts.  These  include 
structured  processing,  standard  data  element  usage,  uniform 
standards,  methods  and  procedures,  improved  programming 
technigues  and  training.  When  all  of  these  various  layers 
are  utilized  and  implememted  as  a  whole  we  have  the  philo- 
sophy  of    FMSO  towards   producing   a   quality    product. 

C.       QUALITY    ASSOHANCE    VS.    QOALITT    CONTHOL 

Quality  Assurance,  as  defined  by  FMSO  [Ref.  UO],  is  a 
line  management  responsibil  i":  y.  As  such.  Line  Supervi^czs 
are  accountable  for  enforcing  the  application  of  standard 
procedures  that  have  been  developed  for  the  primary  purpose 
of  insuring  accuracy,  thoroughness  of  method,  simplici-y  in 
design,  adeguacy  of  testing  and  clarity  of  documentation  of 
ADS  development.  To  aid  all  levels  of  personnel  within  ths 
various  line  departments  in  the  accomplishment  of  their  own 
specific  requirements  for  quality,  "guidelines  have  been 
developed  that  include  NAVSU?  PQBs  506,  507,  508,  CDA 
DEVELOPMENT  HANDBOOK,  CDA  MANAGEMENT  HANDBOOK,  7MS0  In-ernal 
Instructions,  and  various  o-^her  documented  and  undocumented 
departmental  procedures.  Everything  written  and  documented 
must  be  in  compliance  with  these  standards.  It  is  the  indi- 
vidual person's  responsibility  to  ensure  that  they  are  in 
compliance    with    these  standards. 

Quality  Control,  as  defined  by  FMSO  [Ref.  40],  is  the 
responsibility  of  the  Management  Department  and  specifically 
the  Quality  Control  Branch.  Quality  control  procedures  are 
those  actions  that  are  taks n  as  an  ADS  is  being  developed  to 
insure  that  all  the  required  QUALITY  ASSURANCE  procedures  , 
those  actions  performed  by  the  line  department  personnel, 
will  be  complied  with  to  produce  a  r=3liable  and  error  free 
ADS    product.  Quality   Control    than      is   primarily      a   review 
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function  to  be  performed  by  the  Quality  Control  Branch.  In 
this  area  they  are  responsible  for  ensuring  That 
designated/high  interest:  ADS  projects  conform  wich  standards 
of  completeness,  accuracy,  clarity,  and  all  other  applicable 
quality  standards  that  applied  to  thr  line  quality  assurance 
program   have   been  achieved, 

D.       SPECIFIC   QUALITY    CONTROL    RESPOHSIBILITIES 

As  quality  control  is  a  review  function  and  given  the 
limited  number  cf  personnel  assign?!  to  the  branch,  not 
every  cutou*  t h ^t  is  orovided  by  F1S3  will  be  reviewed  by 
the  Quality  Control  Branch.  The  Quali-y  Control  Branch 
will,  however,  be  directly  involved  with  those  projects 
designated  high  priority  or  of  special  interest  to  the 
Command.  All  other  projects  will  be  reviewed  on  a  priority 
basis  as  the  manhours  that  can  be  devoted  to  the  futher 
enhancement  of  the  quality  effort  become  available.  Their 
area  of  responsibility  is  enormous  aad  their  tasks  numerous. 
Th9refcr€,  only  areas  of  major  responsibilities  are  listed 
below   [Kef.    40], 

1.  Review  of  Functional  Descriptions  for  compliance  with 
standards . 

2.  Participate  in  a  system  design  review  to  ensure  that 
the  design  has  considered  all  of  the  proposed  system 
requirements. 

3.  Review  of  System  Specifications  for  compliance  with 
standards . 

4.  Review   of   Program   Specifications. 

5.  Review  of  the  Maintenance  Manual,  Operations  Manual, 
and   all   other    applicable    manuals. 

6.  Prepare  an  analysis  of  PTRs  received  as  to  cause  or 
symptom    and    recommend   possible    corrective   action. 

7.  The  Quality  Control  Branch  will  selectively  review 
test  plans  and  tests  for  compliance  with  Quality 
Assurance  guidelines.  In  the  performance  ot  this 
task,  they  may  desk  check  all  applicable  data  or  they 
may  elect  to  attend  the  review  conference  held 
between  the  grogrammer  and  analyst  as  they  discuss 
the   results    or    the   test. 

8.  Review   the  Implementation   Plan. 
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9.  Quality  Control  is  resoonsibla  for  post  implementa- 
tion visits  to  sslsCtsd*  sitss  of  designated " pro j=cts 
to  determine  whether  the  product  released  is" working 
satisfactorily,  meets  the  nesds  of  the  user,  =nd  is 
being   utilized  correctly. 

10.  Perform  a  Quality  Assurance  Rsview  of  all  designated 
ADS    programs. 


E.       TESTING    TO    ENSURE   QOALITY 

The  testing  process  involves  many  different  personnel 
and  includes  many  different  responsibility  assignments.  For 
example,  the  individual  programmer  assigned  to  -he  project 
is  responsible  fcr  reasonable  testing  for  all  error  condi- 
tions that  could  occur  in  the  program  and  for  providing 
support  for  the  systems  test  required  for  the  application. 
Tha  Lead  Program  rner  is  responsible  for  developing  the  'est 
plan    for   system    testing   and/or  string    testing. 

The  Systems  Analyst  is  responsible  for  assisting  the 
Lead  Programmer  in  planning  and  ooordinating  the  string 
testing/system  testing  to  determine  that  all  the  programs  in 
the  application  produce  the  required  output  when  run  in 
total-  The  Systems  Analyst  has  a  primary  responsibility  of 
approving  and/or  selecting  the  test  data  used  for  systems 
testing. 

The  Systems  Designer  is  to  participate  in  reviews  of 
test  output  to  insure  that  testing  of  the  ADP  program  has 
been  adaquete.  After  the  processes  above  are  completed,  the 
Quality  Control  Branch  will  selectively  review  test  plans 
and  test  results  for  compliance  with  all  Quality  Assurance 
Guidelines. 

1 •      Types   Of   Testing 

During    the      testing    cycle      there   are      primarily    four 
different    methods  utilized    to   check    the    programs. 

1.      Onit   Test   -      A   single    program    that    is      checked    by    the 
programmer  responsible   for   also    writting    the  code.       A 
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Ur.it  Test  Review  is  scheduled  between  the  anaiys-:  and 
programmer  with  ths  test  results  being  reviewed  to 
ascertain    if    futher    testing    is    warranted, 

2,  String  Test  -  Each  program  release  which  requires  the 
execution  of  other  programs  in  actual  production  will 
be  string  tested.  Programmers  are  responsible  for 
writing  the  test  plan  to  ensure  that  all  major  paths 
and  functions  that  will  utilize  the  new  program  are 
checked. 

3.  System  Test  -  Each  new  ADS  or  major  change  involving 
more  than  one  application/operation  or  package  in  an 
exis-ing  ADS  will  be  suDjects!  to  a  system  test  -hat 
will  evaluate  the  specific  system  as  a  whole.  The 
overall  responsibility  for  the  test  will  be  giver,  to 
the  Lead  Programmer.  The  Quality  Control  Branch  will 
evaluate  the  results  to  ensure  that  the  system  meets 
design   objectives. 

U.  Integrated  Systems  Test  -  This  test  will  be  designed 
to  Z9SZ  all  program  interfaces,  databas*e  in-erfaces, 
and  all  internal  and  external  applicaticns  ror 
correctness  of  data  flow.  The  Lead  Programmer  and 
Lead  CDA  Department  are  responsible  for  preparing  a 
formal  test  plan  and  conducting  the  test.  The  Quality 
Control  Branch  may  or  may  lot  be  assigned  as  an 
overall  monitor  for  this  procedure  but  will  be 
required    to   review  the   results, 

F.       SYSTEM    RELEASE    PBOCEDORES 

Upon  completion  of  -he  required  evaluations  and  for 
specified  projects,  the  Line  Departments  involved  will 
forward  to  the  Quality  Control  Branch  all  applicable  docu^ 
mentation  for  review.  After  this  review  has  been  completed 
the    complete   package    is    sent   back   to    the    CDA    Department    Line 
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Manager.  When  the  Managar  is  satisfied  that  the  oroqram 
meets  ~he  requirements  ani  that  all  quality  assurance  s'^-^z- 
dards  have  been  met,  the  Line  Dlanagei:  will,  by  his  signature 
release  the   program    for   use.  This    will   then   terminate   the 

Systems   Development    Process. 

G-       EVALOATTNG    THE    QO&LITY    PROGRAM 

It  is  important  to  realize  that  at  FMSO,  Quality  Control 
is  not  intimately  concerned  with  the  daily  operations  of  the 
line  departments,-  but  is  concerns!  with  assessing  the 
results    of   the   line    departments.  With   this    perspective    in 

mind,  the  design  and  execution  of  th?  Quality  Assurance  Plan 
is  considered  to  be  an  integral  part  of  -he  production 
process  itself.  Quality  assurance  for  software  consists  of 
the  formal  application  Df  st.andards  and  execution  of 
required  tests,  and  then  the  assessment  of  results. 
Enforcement  of  the  standards  and  the  necessary  adjustments 
of  the  process  is  one  of  the  responsibilities  of  the  Quaii-y 
Control    Branch. 

Although  the  Quali-^.y  C  cntrol  3ranch  is  deeply  embedded 
within  the  Management  Department,  they  have  performed  in  an 
extremely  professional  manner.  When  given  an  adequate 
amount  of  time  and  an  opportunity  to  perform  in  their 
primary  role  as  reviewers,  t.hey  have  always  met  the  chal- 
lenge. But  this  opportunity  to  excel  does  not  always  occur 
as  it  should,  and  FMSO  is  no  different  than  any  other  soft- 
ware producing  organization.  As  the  project  completion  date 
draws  near,  usually  the  first  item  to  be  called  excessive  to 
the      program    is      quality    control.  At      FMSO,      the      project 

manager  is  responsible  for  ensuring  that  enough  time  is 
alloted  during  the  S DP  so  that  Quality  Control  has  a  chance 
to  evaluate  the  program.  This  estimate  of  the  amount  of 
time    it    will   take  to    complete      a    project    is    a   most    difficult 
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one,  but  dates  have  to  be  set  and,  dq  occasicri,  me-.  The 
authors  are  unable  to  cite  specific  examples  of  Qiiality 
Conrrol  being  cut  short,  but  when  ths  amount  of  work  that  is 
to  be  accomplished  and  the  number  of  personnel  assigned  to 
accomplish  it  are  compared,  the  assumption  can  be  made  that 
it  has  occurred  ,  either  as  a  result  Df  internally  or  exter- 
nally generated    pressure   tD    comply   with   a    due   date. 

When  the  value  of  a  product  is  very  high,  such  as  on 
command  designated  /special  interest  projects,  each  item 
produced  is  individually  inspected  and  gone  over  in  fine 
detail  at  all  levels,  from  the  programmer  through  all  the 
reviews,  and  finally  top  lanagemer.t.  However,  ur. ler  It-ss 
important  conditions,  the  selection,  and  review  process  of 
products  is  not  as  critical,  and  for  very  minor  projec+:s  it 
need  not  be.  Ihis  does  not  mean  that  the  product  is  any 
less  important  to  the  user  but  merely  implies  that  all 
projects  other  than  the  a xcep tionally  simple  ones  should 
also  receive  the  same  degree  of  scrutiny  that  special 
projects    do. 

The  reality  of  the  present  situation  dictates  that  the 
above  process  is  not  feasible  at  this  time.  With  such  a 
small  staff  in  the  Quality  Control  Branch  and  the  multitude 
of  tasks  assigned,-  it  is  physically  impossible  to  meet  all 
of  their  requirements,  and  priorities  must  be  established. 
With  an  ongoing  review  of  the  number  of  Computer  Specialists 
that  can  be  justified  within  the  Quality  Control  Branch,  it 
is  essential  that  the  justification  be  met  and  billet 
descriptions  constructed  to  place  more  people,  not  less, in 
this  most  important  branch  if  FMS3  is  to  continue  with  its 
present    emphasis   upon   producing    a   quality   product. 

The  authors  feel  that  the  Quality  Control/Quality 
Assurance  program  at  FMSO  is  highly  competitive  with 
similiar  organizations.  Their  thorough  and  most  agressive 
instructions   and   standards    are   excellent,    and   if   the   present 
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level  of  enforcement  is  continued  will  greatly  enhance  "he 
product      serving      the      Fl9=t.  2'--lity      Control/      Quality 

Assurance  does  happen  at  FMSO,  prinarily  due  to  rhe  struc- 
tured process  and  to  the  professionals  that  work,  and  manage 
the    organization.  The    degree      to    which      it   happens      is    an 

evolving   entity. 

At  the  present-  time  tha  only  effective  document.ed  method 
to  measure  the  application  of  guality  practices  within  FMSO 
is  the  analysis  and  evaluation  of  the  Program  Trouble 
Reports  (PTR) .  FTRs  may  bs  submitted  during  any  step  of  the 
systems  development  process  or  after  the  program  has  been 
sen"  tc  the  fi'^ld  user.  \s  a  ^T^  1"  r^c^ived  2.t  F^ISO  it  is 
routed  tc  the  Project  Control  Branch  where  is  is  logged  in 
and  then  sent  to  the  department  that  issued  the  program  with 
which  the  PTR  is  concerned.  The  d=partment  involved  then 
decides  if  the  PTR  is  of  a  crirical  or  non-crirical  nature 
and  then  proceeds  to  work  on  it.  PTRs  are  classified  in 
two  ways,  critical,  which  has  a  significant:  impact  upon 
daily  routine,  and  non-critical,  whirh  has  less  of  an  impact 
and      can    temporarily      be    delayed.  Critical      PTRs    will      be 

corrected  as  socn  as  possible  and  normally  within  three 
working  days  from  receipt  of  sufficiant  information  to  allow 
ths  CDA  to  act.  Non-critical  PTRs  will  be  corrected  as  soon 
as  possible  after  receipt:  of  sufficient  data  that  will  allow 
the    CDA   to    act. 

The  Quality  Control  Branch  performs  a  quarterly  analysis 
of  all  PTRs  received  with  special  emphasis  on  the  most 
common  type  of  problem,  which  steps  of  the  systems  develop- 
ment process  create  the  most  errDrs,  identification  of 
trsnds,  and  if  possible,  recommendations  for  corrective 
actions, 

A  survey  of  the  PTR  Analysis  Report  for  the  Second 
Quarter   CY   82  reveals   the   following:      [Ref.    41], 
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1.  A   total    of  385   PTRs,    of  which    275    were  complt^ted,      92 

were  due  to  either  being  invalid  or  already  in  exis- 
tence. A  reclassified  PTR  was  one  -har,,  when 
submitted,  it  was  felt  by  the  user  that  the  program 
was  not  performing  as  desired  or  requested  but  upon 
researching  the  problem  it  was  found  that  all 
requirements  had  been  met.  To  meet  the  new  require- 
ment of  the  user  a  new  project  would  have  to  be 
designated. 

2.  Of  the  coirpleted  PTRs,  38  were  critical  and  237  non- 
critical  . 

3.  The  FMSO  average  number  of  manhours  to  fix  a  PTR  was 
16  for  a  critical  PTR  and  21  for  a  non-critical  PTR. 
One  possible  reason  for  the  difference  in  times  is 
the  experience  level  of  personnel  assigned  to  repair 
a  program.  A  mor?  experienced  programmer  is  gener- 
ally   assigned    to   a   critical   PTR. 

u.  The  average  number  of  days  for  FMSO  to  complete  a 
critical  PTR  is  15.5  and  151.6  to  complete  a  non- 
critical  PTR.  This  figure  may  be  skewed  toward  the 
high  side  very  easily  because  of  the  small  number  of 
critical    PTRs    but    does    bear   close   monitoring. 

5.  Of  the  completed  PTRs,  57  percent  were  caused  by 
coding/design  errors.  2  1  percent  were  classified  as 
other  and  not  designated  to  any  category.  However, 
42  percent  of  the  critical  PTP.s  were  caused  by 
program  or  coding  errors.  Steps  have  been  taken  to 
better  divide  the  cause  category  in  an  attempt  to 
better  evaluate  the  errors  that  were  classified  in 
the   other   category. 

6.  The  comparative  completion  rates  for  critical  PTRs 
has  remained  relatively  stable  compared  to  the  past 
year.        However,    the    non-critical   completion   rate   has 
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increased   substan-ii lly    and    zh^    -rend      is    fcr    in    ^v^n 
greater   completion   rate. 

7,  The  Received-'Resolvad-Outstaniing  results  show  rhat 
the  number  of  PTRs  received  is  leveling  off,  the 
number  of  PTP.s  resolved  is  increasing  drastically, 
and  as  a  result  of  both  of  these  the  number  of 
outstanding   PTRs   is    now   in   a    steady   decline. 

8.  The  number  of  programs  released  increased  about  12 
percent  over  the  prsvious  quarter  and  yet  the  number 
of  PTRs  received  decreased  by  about  7  percent. 
Although  the  number  of  PTRs  over  the  last  year  has 
oscillated,  the  trend  shows  th=re  to  bs  no  signifi- 
cant increase  and  thus  a  net  decrease  in  ratio  of 
programs    to   PTRs    should    be    anticipated. 

Considering  that  only  a ine  personnel  are  assigned  to  the 
Quality  Con-^.rol  Branch  and  that  FMSO  now  has  10,000  plus 
programs  in  existence,  tha  Quality  Control/Quality  Assurance 
plan  appears  to  be  headed  in  the  rig-ht  direc-^ion  as  the  PTR 
report      clearly      shows.  With    con-cinued      emphasis      on      the 

Quality  Control  effort  an  even  more  effective  program  will 
be  displayed  in  the  futura.  To  a::hiave  a  100  percent  error 
frae  product  is  the  ideal  but  a  mora  realistic  goal  must  be 
set  and  an  effective  method  of  measuring  and  selecting  the 
programs  for  a  more  detailad  investigation  will  aid  greatly 
in   accomplishing      the  goal    FMSD      sets    for    itself.  The   PTR 

report  shows  an  improvement  over  tha  preceeding  year  but  it 
also  quite  effectively  shows  other  areas  that  may  need  addi- 
tional emphasis.  In  the  following  chapter  we  will  list  some 
of  -he  areas  that  we  feel  should  be  considered  in  the  future 
corporate   growth   pattern   of    FMSO. 
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7.  RSCOMMENDATIDMS 

This  paper  has  been  a  comprehBiisive  investigation  of 
software  quality  assurance  from  a  theoretical  viewpoint  and 
from  specific  investigation  into  the  quality  assurance/ 
quality  control  departments  of  variois  organizations.  Based 
on  the  authors'  research,  the  following  recommendations  are 
offered  in  hopes  of  enhancing  the  Fleet  Material  Support 
Office's  (FMSO)  quality  control  effort. 

1.  \  corporate  base  line  concer?.ing  th^  quality  prcc='S3 
needs  to  te  established.  Before  a  specific  direction 
can  be  maintained,  a  mechanism  must  exist  that  would 
enable  FMSO  to  msasure  its  ability  to  meet  the 
desired  obj-ectives.  It  is  fslt  that,  at  a  minimum, 
the  Quality  Control  Branch  could  examine  past 
programs  and  select  at  least  two  that  were  very 
similar  for  an  analysis  as  to  the  effectiveness  of 
the  quality  conrrol  program.  The  cri-ceria  of  ihes- 
two  programs  might  be  (1)  that  they  were  produced  by 
the  same  departmeat  within  a  short  time  period  of 
each  other,  (2)  that  they  were  intended  to  be 
utilized  in  the  same  fashion,  and  (3)  that  one  of 
them  had  been  reviewed  by  Quality  Control  throughout 
the  entire  process  and  the  other  had  not  received 
this  same  critical  review. 

2.  Top  management  must  continus  to  re-emphasize  the 
importance  of  quality  control  within  the  organization 
and  display  a  positive  philosophy  of  commitment  to 
the  quality  process.  ks  has  been  discussed,  without 
top  management  support,  quality  assurance/quality 
control  programs  produce  a  less  than  desirable 
output. 
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The  number  of  personnsl  assigned  to  the  quality 
control  trar.ch  is  defini+-?ly  inadequate.  In  an 
organization  as  large  as  FMSO,  which  produces  800 
plus  programs  per  quarter,  a  quality  control  organi- 
zation of  only  eight  people  plus  a  supervisor  can  not 
reasonably  be  expected  to  mast  their  obligations  and 
responsibilities  on  all  occasions.  The  Quality 
Control  Branch  nssds  an  infusion  of  personnel. 
Attention  should  be  paid  not  only  to  the  quantity  of 
personnel  but  also  to  quality.  As  stated  previously, 
the  core  of  these  personnel  need  to  be  as  knowledge- 
able as  senior  systems  analys-.s  and  also  command  the 
respect  of  the  individuals  whose  systems  are  being 
evaluated . 

A  method  of  augmenting  ^he  s-aff  of  the  Quali-'-y 
Control  Branch  from  an  external  source  may  be  to 
utilize  it  as  an  indoctrination  facility  for  new 
hires.  A  core  of  highly  qualified  personnel  could  be 
maintained  .that  were  the  permanent  part  of  the 
Quality  Ccntrcl  Branch.  Hew  hires  could  be  given 
this  temporary  position  and  tasked  with  such  projects 
as  reviewin-g  the  project  sp  =  :::if ica-ions,  all  of  the 
manuals,  and  all  other  applicable  material.  This 
would  also  afford  zhe  new  personnel  an  opportunity  to 
receive  formal  and  correct  training  before  being 
assigned  to  the  particular  line  department. 
Additionally,  it  would  enable  the  new  hires  to  gain  a 
better  overall  understanding  of.  what  the  organization 
does  and  the  amount  of  interfacing  that  must 
conducted  before  a  project  is  completed.  Of  course 
there  would  have  to  be  some  discretionary  measures 
imposed  when  selecting  personnel  and  a  time  limit 
must  be  adhered  to. 


109 


5.  Ccnsideration  shDuli  be  given  to  moving  ^he  Quality 
Control  Branch  out  cf  its  pu=seat.  managemen-  s-ruc- 
tur?.  T hs  authors  f"??!  that,  because  of  th'5  i^.assiv? 
size  of  t-he  organization  and  enormous  amount  of 
material  that  musr  be  reviewed,  the  Quality  Control 
Branch  should  be  moved  so  as  to  have  at  least  parity 
with  line  management.  This  position  may  be  desig- 
nated as  Quality  Control  Department  Head,  Code  99. 
Billet  descriptions  would  have  to  be  re-written  and 
the  management  problems  overcome,  but  the  increased 
communications,  be  they  voluntary  or  by  direction, 
between  qualify  coriuroi  and  line  aiar-agemeni:  ^ouli 
have  an  impact  upon  future  products.  Additionally, 
this  would  afford  the  qualit/  control  personnel  an 
easier  avenue  to  bscome  more  directly  involved  with 
their  responsibility  of  ensuring  that  standards  are 
met  and  better  snable  them  to  enforce  these 
standards , 

6.  Line  managers  must  be  educated  as  to  the  importance 
of  the  quality  control  function.  In  this  vein,  when 
openings  arise  in  the  Quality  Control  Branch,  quali- 
fied line  personnel  should  be  encouraged  to  apply  for 
the  positions.  The  benefit  gained  for  the  organiza- 
tion would  more  than  offset  the  loss  to  one 
in dividua 1  department.  To  support  the  attitudinal 
change  that  must  take  place,  an  ongoing  training 
program,  sponsored  by  top  management,  will  have  to  be 
implemented  on  a  company  wide  basis  and  the  positive 
benefits  gained  from  the  addition  of  these  qualified 
personnel  to  the  Quality  Control  Branch  must  be 
discussed  and    displayed. 

7.  Quality  control  checklists  must  be  re-insxituted. 
There  should  be  a  general  checklist  applicable  to  all 
programming    functions   as    well      as    specific   checklists 


110 


regarding  each  iniividaal  prograni.  I-^.  should  be 
emphasized  that  t.hase  checklists  are  to  be  used  as 
reminders  or  as  a  constructive  tool  to  reinforce 
standards ,  .rather  than  a  method  of  laying  blame. 

8.  It  is  important  for  continuity  that  there  be  consis- 
tent assignment  of  guality  control  personnel  to 
projects.  One  person  (or  team  of  persons)  should  be 
assigned  to  a  project,  except  the  smallest  ones,  and 
should  monitor  this  project  all  the  way  from  incep- 
tion to  post  implementation  review. 

9.  A  better  iiethod,  possibly  a  decision  support  system, 
musx  be  devised  to  deteraiiae  which  projects  will 
undergo  quality  control  procedures.  It  is  recognized 
-hat  the  number  of  projects  may  preclude  all  projects 
being  scrutinized  by  quality  control.  However,  there 
needs  to  be  a  better  system  to  select  programs  for 
review  and  evaluation  than  the  present  method  of 
committee  or  command  discretion.  Currently,  the 
primary  method  for  evaluating  project  length  and 
complexity  is  experience.  This  subjective  viewpoint 
is  the  basis  for  the  depth  of  involvement  achieved  by 
quality  control.  A  logical  and  comprehensive  deci- 
sion process  will  serve  to  alleviate  crisis 
management  and  ensure  that  the  most  important 
projects  are  chosen. 

10.  End  users  must  become  even  more  involved  in  the 
design  and  development  of  software  throughout  the 
lifecycle.  Users  should  work  closely  with  quality 
control  to  ensurs  a  timely  and  correct  review 
process.  There  nust  also  be  better  communications 
established  between  users  and  line  programming  func- 
tions. This  communication  may  be  facilitated  by 
quality  control  and  may  help  lower  project  trouble 
report  (?TR)  reclassifications. 
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11.  clear  and  concise  io  cumentatiDn  is  an  ongoina  problatn 
at  any  software  production  facility.  The  quality 
control  division  oiust  review  documentation,  not 
coding,  and  assist  where  possible  to  improve  any 
deficiencies. 

12.  Currently,  the  program  trouble  reports  (PTR) ,  are  rhe 
primary  documented  measure  of  the  effectiveness  of 
the  guality  program.  Whils  this  may  be  a  valid 
measurement,  there  needs  to  be  a  search  for  addi- 
tional measures.  Since  quality  control  does  not 
impact  100  percent  of  the  programs  leaving  FMSO,  a 
fliethod  of  evaluating  the  effect  of  quality  ccn-rcl  on 
projects  which  do  no*  have  quality  control  involve- 
ment   must   be    formulated. 

13.  The  principle  of  top-down  design  and  top-down  testing 
should  be  reevaluated.  As  an  altermative,  top-down 
design  and  bottom  up  testing  should  be  considered. 
It  is  assumed  that  top  ofianageien t  will  feel  uneasy  as 
The  present  process  'is  ohangei.  However,  it  has  been 
shown  in  many  studies  "hat  errors  niade  in  the  design 
phase  of  a  project  are  the  mDSt  expensive  to  repair 
because  one  has  to  return  back  to  the  beginning  and 
literally  begin  the  entire  process  over. 
Requirements  must  be  reevaluated,  specifications 
redone,  coding  rewritten,  and  finally,  the  program 
must  be  tested  again.  Since  the  majority  of  design 
errors  are  not  found  until  the  testing  phase,  it  is 
easy  to  see  that  the  amount  of  extra  time  spent  in 
the  design  phase  will,  over  time,  offset  any  anxiety 
that  top  management  would  have  because  of  the  seem- 
ingly laclc  of  progress  on  the  project.  It  is 
suggested  that  bottom-up  tasting  will  force  the 
design   effort    to   improve. 
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14,  Phased  developaent  should  be  adhered  to.  Coding  laust 
net  begin  until  after  the  dssign  is  comple-^.  3y 
doing  sc  the  lead  programmer  can  assign  the  bet-:er 
programmers  to  the  difficult  aodules  and  an  inexperi- 
enced programmer  to  the  easier  modules/programs. 
This  line  of  thought  should  filter  throughout  the 
project  and  allow  for  better  personnel  utilization, 
but  should  also  afford  the  persons  writing  the  test 
plan  to  do  so  in  a  lore  reliable  manner.  In  allowing 
the  writing  of  code  before  the  final  design  is 
completed,  •  we  achiave  the  short  term  goal  of  being 
able  to  see  a  worKing  product,  -cha-^  can  be  tracked  on 
a  chart.  However,  we  cannot  see  the  long  range  goal 
of  whether  the  module  will  produce  the  required 
output   or   the    required  number   of    interfaces. 

15.  The  system  to  trace  the  phase  of  development  for  each 
project  should  be  reemphasized  and  quality  control 
must  continue  to  05  kept  abraast  of  all  the  current 
production  efforts. 
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